No title

REGULARITY PROPERTIES OF A FREE BOUNDARY NEAR CONTACT POINTS WITH THE FIXED BOUNDARY HENRIK SHAHGHOLIAN and NINA URALTSEVA

Abstract In the upper half of the unit ball B + = {|x| < 1, x1 > 0}, let u and  (a domain in Rn+ = {x ∈ Rn : x1 > 0} ) solve the following overdetermined problem: 1u = χ

in B + ,

u = |∇u| = 0 in B + \ ,

u=0

on 5 ∩ B,

where B is the unit ball with center at the origin, χ denotes the characteristic function of , 5 = {x1 = 0}, n ≥ 2, and the equation is satisfied in the sense of distributions. We show (among other things) that if the origin is a contact point of the free boundary, that is, if u(0) = |∇u(0)| = 0, then ∂ ∩ Br0 is the graph of a C 1 -function over 5 ∩ Br0 . The C 1 -norm depends on the dimension and sup B + |u|. The result is extended to general subdomains of the unit ball with C 3 -boundary. 1. Introduction The regularity properties of solutions to a certain type of free boundary problem are the main object of study in this paper. Mathematically the problem is formulated as follows. Let  ⊂ Rn+ , and suppose that there is a function u solving the following problem: 1u = χ

in B + ,

u = |∇u| = 0

in B + \ ,

u=0

on 5 ∩ B, (1.1)

where B + is the upper half of the unit ball with center at the origin, and 5 = {x1 = 0} (see Fig. 1). Let us denote the free boundary {x : u(x) = |∇u(x)| = 0} ∩ ∂ by 0(u). The behavior of the free boundary near the contact points 00 (u) = 0(u) ∩ 5 is of main interest in this paper. DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 1, Received 31 May 2001. Revision received 2 November 2001. 2000 Mathematics Subject Classification. Primary 35R35, 35J60. Shahgholian partially supported by the Swedish Natural Sciences Research Council. Uraltseva supported by Russian Foundation of Fundamental Research grant number 99-0100684. 1

2

SHAHGHOLIAN and URALTSEVA

0  1u = χ

3(u) = {u = |∇u| = 0}

5 00 = contact points Figure 1

This problem was considered earlier by N. Uraltseva [U1] and D. Apushkinskaya and Uraltseva [AU] but with restriction on the sign of the solution. Under the Lipschitz regularity assumption on ∂, the authors in [AU] could prove that the free and the fixed boundary meet tangentially. Later, it was shown by [U1] that the 0(u) is C 1 . Recently, Uraltseva has extended these results to the situation with no sign restriction on u and with a certain thickness assumption on the set 5 ∩ {|∇u| = 0} (see [U3]). For the case of u ≥ 0, any a priori assumptions on ∂ were removed in [U2]. Our main objective in this paper is to study the problems with neither sign restriction nor Lipschitz regularity assumptions. We still conclude that the free boundary is a graph of a C 1 -function. This is now possible due to recent progress in related problems (see [CKS]), using strong tools such as the monotonicity formula (see also [Ca1] – [Ca3] for background and developments in the case of the obstacle problem). Motivation. Our motivation for studying this problem is threefold. The motivation for studying contact points is related to problems in mathematical physics, where certain data produce contact points between the free (unknown) and the prescribed boundary. Such problems appear in filtration (see [F]), in motion by mean curvature with nonconvex obstacles, and in one- or two-phase problems in flame combustion describing the propagation of curved premixed flame when the fixed and the free boundary touch (see [BL], [BLN], [BCN], [Gu]). The second reason for studying this problem is the peculiarity of the “no-sign assumption” on the solution. Since, generally, solutions to free boundary problems are physical quantities and such quantities are positive in most applications, it may occur that our problem is artificially created. However, in purely mathematical (read academic) problems such as the “inverse of the Cauchy-Kowalewski theorem” or

REGULARITY PROPERTIES OF A FREE BOUNDARY

3

“harmonic continuation of potentials” (see [KS], [CKS]) and in some applications such as chemical reaction interface (see [CH]), the two-phase Bernoulli problem (see [ACF]), or problems involving a system (see [KSt, Chap. VI, Secs. 5, 6]; here one takes u = u 1 − u 2 , where u 1 , u 2 are solutions of the system), it may be the case that there is no restriction on the sign of the solution. In this connection we also refer to [ACS] for a problem in inverse potential theory where the no-sign assumption is dictated by the problem. Yet another problem that has this special feature of the no-sign assumption is a free boundary arising in the stationary case in the mean-field theory of superconductivity (see [ESS], [CS]). Our third, and maybe the most important, reason for this study is to develop a technique that handles problems of the above nature. This was initiated in [KS] and further investigated in [CKS]. The advantage of the technique presented in this paper, besides its flexibility, is its simplicity and clarity. The local fine analysis used in the (by now) classical approach of [Ca1], [Ca2], and [U1] has been abandoned in favor of the global analysis of [Ca3] and [CKS]. Notation. We use the following notation throughout the paper: C, C0 , C1 χD D ∂D |D| x, X Rn+ , Rn− Br (x), B(x, r ) Br+ (x) Br , Br+ 3(u) (u) 5 0(u) ˜ 3(u) 00 (u) 3r (u, x) Proj(V1 , V2 ) ˜ r (u, x) 3 k · k∞ MD e1 , . . . , en

generic constants; the characteristic function of the set D (D ⊂ Rn , n ≥ 2); the closure of D; the boundary of a set D; n-dimensional volume of the set D; x = (x1 , . . . , xn ), X = (x2 , . . . , xn ); {x ∈ Rn : x1 > 0}, {x ∈ Rn : x1 < 0}; {y ∈ Rn : |y − x| < r }; Br (x) ∩ Rn+ ; Br (0), Br+ (0); {x : u(x) = |∇u(x)| = 0} for any C 1 -function u; Rn+ \ 3(u); {x : x1 = 0}; ∂(u) ∩ 3(u); free boundary; projection of 3(u) on 5; 0(u) ∩ 5 contact points; 3(u) ∩ Br (x); projection of the set V1 onto V2 ; Proj(3r (u, x), 5); supremum norm; minimum diameter (see Def. 1.2); standard basis in Rn ;

4

SHAHGHOLIAN and URALTSEVA

ν, e Dν , Dνe e⊥ν Pr+ (M), . . . v+, v−

arbitrary unit vectors; first and second directional derivatives; e is orthogonal to ν; (see Def. 1.1); max(v, 0); max(−v, 0).

In order to state our main results, let us define a local solution. Definition 1.1 We say a function u (not identically zero) belongs to the class Pr+ (M) if u satisfies (in the sense of distributions) (1) 1u = χ in Br+ for some open set  ⊂ Rn+ , (2) u = |∇u| = 0 in Br+ \ , (3) kuk∞,Br+ ≤ M, (4) u = 0 on 5 ∩ Br . The interior points of the free boundary are well studied in [CKS]. The interior problem is analogous to the one in which Br+0 (0) is replaced by Br0 (0) and u solves the problem above (without (4)) in the interior of Br0 (0) (and without the boundary condition on 5). In this paper we consider the free boundary near and on 5. Observe that the class Pr+ (M), as defined above, is invariant under rotation in 5, that is, rotations that leave e1 fixed. Next we define a subclass of Pr+ (M) which is the main class under consideration in this paper, namely,  Pr+ (0, M) := u ∈ Pr+ (M) : 0 ∈ 0(u) . + (M), P + (0, M) the corresponding “global solutions” with We also denote by P∞ ∞ quadratic growth, that is, solutions in the entire upper half-space Rn+ with quadratic growth |u(x)| ≤ M(|x| + 1)2 . We define similarly Pr (M), Pr (0, M), and P∞ (0, M), where we replace Br+ and Rn+ by Br and Rn , respectively, and  is an open set in Rn . Here there is no extra Dirichlet data assumed on the plane 5 (see [CKS]). Examples of global solutions can be given easily. For a ≥ 0,

((x1 − a)+ )2 2 is a global solution. When a = 0, the free boundary coincides with the fixed boundary, and for a > 0 the free boundary is {x1 = a}. Observe that the solution here is onedimensional. + (0, M) can be represented by Two-dimensional solutions in P∞ u(x) =

n

u(x) =

x12 X + ai x1 xi . 2 i=2

REGULARITY PROPERTIES OF A FREE BOUNDARY

5

Here the free boundary is of dimension (n − 2) if ai 6= 0 for some i ≥ 2. One of the main results of this paper asserts that these are the only global solutions (see Th. B). Examples of local solutions are harder to give since one has to construct them. However, there are some classical examples for the interior free boundaries, that is, for the problem inside the unit ball. Fortunately, some of these examples can be used to produce examples of contact points. For clarity and for the reader’s convenience, we give some details of this procedure. Suppose n = 2, and recall from [KN, pp. 387 – 390] that there are examples of the free boundary in the interior of the ball, where cusps appear. These cusps are represented by the curves µ/2

x2 = ±x1 ,

0 ≤ x1 ≤ 1,

where µ = 4k + 1 (k = 1, 2, . . .) gives nonnegative solutions and µ = 4k + 3 (k = 0, 1, . . .) gives solutions that become negative on the negative x1 -axis. The solution is defined locally by  2 µ u(x) = x22 − ρ 1+µ/2 sin 1 + θ + · · · , x ∈ , |x| < ε, 1 + µ/2 2 for ε small. Here we use both real and complex notation x = (x1 , x2 ),

z = ρeiθ ,

0 ≤ θ ≤ 2π.

Also, the domain  is the image of the set {z : |z| < 1, Im z > 0} under the conformal mapping f (z) = z 2 + i z µ . Now for the case of µ = 4k +3, the solutions are negative on the negative x1 -axis. However, one can show that the set ∂{u < 0} consists of two C ∞ -curves, one above and one below the negative x1 -axis, and they meet the x1 -axis tangentially. (We do not go into the details of this part; for the specific case of the cardioid, one can even compute this numerically.) Let us denote these graphs by f 1 , f 2 , respectively. We also assume that these graphs are identically zero on the positive x1 -axis. Now consider a conformal mapping, which takes the lower part, that is, f 2 , into the x1 -axis. Then our new free boundary is exactly the one that hits the x1 -axis tangentially, and it lies above the positive x1 -axis. The set where {u < 0} lies above the negative x1 -axis, and it has zero Lebesgue density at the origin. Moreover, our transferred function is in the class Pε+ (0, M) for some small ε. By scaling we may transfer the solution to the unit upper half-ball. Definition 1.2 requires the following notation:
˜ r (u, x) = Proj 3r (u, x), 5 , 3r (u, x) = 3(u) ∩ Br (x), 3

6

SHAHGHOLIAN and URALTSEVA

where Proj(3r (u, x), 5) means the projection of 3r (u, x) onto 5. We suppress the ˜ r (u, x) when it is the origin; otherwise, there is no ambigupoint x in 3r (u, x) and 3 ity. Definition 1.2 (Minimal diameter) The minimum diameter of a bounded set D ⊂ Rk , denoted MD(D), is the infimum of distances between pairs of parallel hyperplanes in Rk such that D is contained in the strip determined by the planes. We also define the density function δr (u, x) =

˜ r (u, x)) MD(3 . r

˜ r we have 3 ˜ r ⊂ 5 ≈ Rn−1 ; that is, k = n − 1 in Observe that by our definition of 3 this case. Let us state the main results of this paper, that is, Theorems A, B, C, and D. A There is a constant C1 (depending only on the dimension) such that if u ∈ P1+ (M), then sup |Di j u| ≤ C1 M. THEOREM

B + (0,1/2)

The proof of Theorem A follows the main steps in that of [CKS, Th. I]. We mention only the minor changes of the new situation later in Section 2. THEOREM B + (M); then the following hold: Let u ∈ P∞

x12 + ax1 x2 + αx1 2 in some rotated system of 5, and for some real numbers a and α, ∂ ∩ {x1 > 0} = ∅ =⇒ u(x) =

∂ ∩ {x1 > 0} 6= ∅ =⇒ u(x) =

((x1 − a)+ )2 2

for a > 0.

+ (0, M) and, consequently, Observe that if the origin is a contact point, then u ∈ P∞ + (M) have α = 0 in Theorem B. Also, recall that by Definition 1.1 members of P∞ quadratic growth at ∞. Before stating the next result, we need to define the so-called singular set.

Definition 1.3 (Singular set) A point x 0 is said to be a singular contact point if x 0 ∈ 00 (u), and there is a blow-up

REGULARITY PROPERTIES OF A FREE BOUNDARY

7

(see Sec. 3) u 0 of u at x 0 which, after translation and rotation, has the representation u0 =

x12 + ax1 x2 2

in Rn+ (a 6= 0).

(1.2)

For definiteness we use the notation Su (a0 ) to denote the set of all singular points such that the blow-up of u at the singular point has the representation (1.2) for some |a| ≥ a0 . x0

The following properties can be listed for the free boundary 0(u) and the solution u. C There exist a universal constant r0 = r0 (n, M) and a modulus of continuity σ (σ (0+ ) = 0) such that if u ∈ P1+ (0, M), then the following hold. (C1) We have  ∂ ∩ Br0 ⊂ x : x1 ≤ σ (|x|)|x| . THEOREM

(C2)

(C3) (C4) (C5)

∂ ∩ Br0 is the graph of a C 1 -function over 5 ∩ Br0 . The C 1 -norm is uniform for the class. More exactly, the modulus of continuity of the normal vector to the boundary of  is σ (r ). The singular set Su (a) of ∂(u) lies locally in an (n − 2)-dimensional C 1 manifold 50 ⊂ 5. The neighborhood depends on the constant a in (1.2). + We have that limx→∂(u) Di j u exists nontangentially for x ∈ (u) ∩ B1/2 . If δr (u, 0) > σ (r ) for some r < r0 , then sup Br+

(C6)

|De u(x)| ≤ σ (r ) x1

(1.3)

for all e orthogonal to e1 . If n = 2, then Di j u ∈ C((u) ∩ Br0 ). If n > 2, then Di j u are continuous in ((u) ∩ Br+0 ) ∪ {0}, provided limr →0 δr (u, 0) > 0.

For n = 2, the statement of (C3) means that singular contact points are isolated, while (C5) simply asserts that if the origin is a free boundary point and if there is a free boundary point x 0 that comes too near the origin, then the conclusion of Theorem C holds. Observe that Theorem C is point independent; that is, the same statements hold for all contact points in B1/2 . The singular set of the interior points of the free boundary has been studied by L. Caffarelli [Ca3] for the case of u ≥ 0, and by Caffarelli and H. Shahgholian [CSh] for the general case. In [Ca3], using the nonnegativity of the solutions, one can give a detailed description of the singular set in terms of the blow-up solutions. Notably,

8

SHAHGHOLIAN and URALTSEVA

0  1u = χ

3(u) = {u = |∇u| = 0} G

00 = contact points Figure 2

Caffarelli’s technique for studying the structure of the singular set seems to fail for the interior singular points when there is no sign restriction on the solution. However, in [CSh] the authors are able to prove that the singular set locally lies on a C 1 -manifold of dimension k, where k is the dimension of the linear subspace 3(u 0 ). A similar analysis works here as well. For the next result we refer to Figure 2. THEOREM D All the results in Theorems A and C can be generalized to hold when 5 is replaced by a C 3 -surface. More exactly, we assume that there is a C 3 -graph G 0 containing the − origin and dividing the unit ball B1 into two parts, G + 0 and G 0 . On one side of this + + graph, G 0 , say, we assume that u and  (⊂ G 0 ) solve (1.1) with B + replaced by G+ 0 . Then there exist a small constant r0 and a modulus of continuity σ (r ), depending on n, M, and the C 3 -norm of G 0 , such that if 0 ∈ G 0 ∩ 0(u) with u satisfying the conditions in Definition 1.1 (with the appropriate changes), then all the results above (Ths. A and C) remain valid in this case. The quantities naturally also depend on the C 3 -norm of the surface G 0 .

Parts (1) and (2) in Theorem C for the general case, that is, Theorem D, should be interpreted in terms of the tangent plane of G 0 at the origin. A fundamental tool for proving our theorems is the following monotonicity lemma. LEMMA 1.4 ([ACF, Lemma 5.1]) Let h 1 , h 2 be two nonnegative continuous subsolutions of 1u = 0 in B(x 0 , R) (R >

REGULARITY PROPERTIES OF A FREE BOUNDARY

9

0). Assume further that h 1 h 2 = 0 and that h 1 (x 0 ) = h 2 (x 0 ) = 0. Then the following function is monotone in r (0 < r < R): Z Z 1 |∇h 1 |2 d x  |∇h 2 |2 d x  ϕ(r ) = ϕ(r, h 1 , h 2 , x 0 ) = 4 . (1.4) 0 n−2 0 n−2 r B(x 0 ,r ) |x − x | B(x 0 ,r ) |x − x | More exactly, if any of the sets supp(h i ) ∩ ∂ Br (x 0 ) digress from a spherical cap by a positive area, then either ϕ 0 (r ) > 0 or ϕ(r ) = 0. In proving Theorem D, we need to use a different type of monotonicity formula, one which is similar to (1.4) but relaxes the subharmonicity condition and allows the solutions to have a bounded Laplacian only from below. LEMMA 1.5 ([CJK, Th. 1.6]) Recall the assumptions in Lemma 1.4, and replace the subharmonicity assumption by the boundedness (from below) of the Laplacian of h i ; that is, assume 1h i ≥ −1. Suppose, moreover, that |h i (x)| ≤ C|x − x 0 |β for some β > 0. Then β

β

ϕ(s1 ) ≤ (1 + s2 )ϕ(s2 ) + Cs2 ,

(1.5)

where 0 < s1 ≤ s2 ≤ R. In Lemma 1.5 if we have 1h i ≥ −Ci , then we can replace h i by h i /Ci and change the constant C in (1.5). The reader may easily verify that any function verifying (1.5) must have a limit as r → 0+ ; that is, lim ϕ(r ) exists. (1.6) r →0+

In the sequel we use the notation ϕ(r, De u) = ϕ(r, De u, x 0 ) in (1.4) and (1.5) with u ∈ P1+ (M) and h 1 , h 2 replaced by (De u)± . Here e ⊥ e1 and x 0 ∈ 5, and De u is continued into {x1 < 0} in a proper way. Lemma 1.5 can be used to generalize Theorems A and C to the case of 1u = f χ with f > 0 and Lipschitz since 1(De u)± are now bounded from below on . Remark 1.6 We would like to stress one crucial point in the use of the monotonicity formulas with respect to the functions u ∈ P + (M). As discussed above, the functions (De u)± are admissible for using the monotonicity formula when e ⊥ e1 , that is, when e is parallel to 5, and x 0 ∈ 5.

10

SHAHGHOLIAN and URALTSEVA

In the applications of this we use three different continuations of De u across the fixed boundary 5. (1) We consider the odd reflection in 5 by setting u(x) = −u(−x1 , x2 , . . . , xn ) for x ∈ Rn− . This is used in connection with the proofs of Theorem A, Theorem B (partially), and Theorem C. (2) We may continue u as zero across 5 to the lower half-space Rn− . This is used (partially) in connection with the proof of Theorem B. (3) In connection with the proof of Theorem D, we consider odd inversion with respect to a sphere (see the proof of Th. D). 2. Proof of Theorem A We extend u ∈ P1+ (M) to the lower half-part of the unit ball by considering the odd reflection of u in the plane 5; that is, we extend u across the plane 5 by defining it as −u(−x1 , x2 , . . .) for x1 < 0 and x in the unit ball. In this way our functions have a bounded Laplacian, and De u, for e orthogonal to e1 , is continuous in the whole unit ball. Moreover, (De u)± are subharmonic. Observe also that in [CKS, Th. I] one uses only the boundedness of the Laplacian of u and the subharmonicity of (De u)± in order to deduce the quadratic behavior of the solution near the free boundary. As in [CKS], we set S j (z, u) = sup |u|, B(z,2− j )

and we define M(z, u) to be the maximal subset of N (natural numbers) satisfying the doubling condition 4S( j+1) (z, u) ≥ S j (z, u),

∀ j ∈ M(z, u).

Following [CKS], we consider the next lemma. LEMMA 2.1 + Let u ∈ P1+ (M) and z ∈ 0(u) ∩ B1/2 . Then there exists a constant C0 = C0 (n) such that S j (z, u) ≤ C0 M2−2 j , ∀ j ∈ M(z, u). (2.1)

Proof Extend u, as in the discussion above, to the whole unit ball. Let e be orthogonal to e1 (x1 -axis), and then follow the steps in [CKS, Lem. 3.2] to end up with De u 0 = 0 for all such directions. Here u 0 is the limit function. This, in particular, implies that u 0 is one-dimensional, and it depends only on the x1 -direction. Since it is also harmonic, it must be linear; that is, u 0 (x) = ax1 + b. But then u 0 (0) = u 00 (0) = 0 implies u 0 ≡ 0. This contradicts [CKS, (3.5)]. Hence (2.1) holds.

REGULARITY PROPERTIES OF A FREE BOUNDARY

11

Observe that the estimate |u(x)| ≤ C Md(x)2 for all x ∈ (u) is equivalent to |D 2 u| ≤ C0 M. This can be proven along the same lines as [CKS, end of Sec. 3] in combination with elliptic estimates near the fixed boundary. We leave the details to the reader. 3. Certain facts In this and the next sections, we frequently use the blow-up of a function at a point x 0 ∈ 0(u); that is, for a given u we consider u r (x) =

u(r x + x 0 ) , r2

and we let r tend to zero, through some subsequence. It is, however, not clear whether the blow-up (the limit function) is not the zero function. Indeed, if u(x) = o(|x − x 0 |2 ), then any blow-up is identically zero. To prevent this, we need a nondegeneracy from below, asserted in (3.1). However, we first remark that, in view of the maximum principle, in any neighborhood of a free boundary point there are points where u takes positive values. Nondegeneracy The function u in (1.1) satisfies sup u ≥ u(x 0 ) + Cn r 2

B + (x 0 ,r )

(3.1)

for all x 0 ∈ (u), where u(x 0 ) ≥ 0, and also for all x 0 ∈ 0(u). Here Cn = 1/2n. Also, r is small enough so that B(x 0 , r ) ⊂ B(0, 1). For Br (x 0 ) ⊂ B1+ , (3.1) is also true if u(x 0 ) < 0 with somewhat smaller Cn . For a proof, see [Ca2] and [CKS]. ε-close We say two functions f and g are ε-close to each other in a domain D if sup | f (x) − g(x)| < ε. x∈D

It can be proven that ε-closeness of u ∈ Pr+ (M) and ψ(x) = ((x1 − a)+ )2 /2 (a > 0) implies √ sup |Du − Dψ| ≤ C ε Br+−√ε

with C = C(n, M). Here we assume r >

√ ε.

12

SHAHGHOLIAN and URALTSEVA

Blow-up limit A blow-up limit u 0 is a uniform limit on compact subsets of a certain subset of Rn (to be specified in each case): u 0 (x) = lim

j→∞

u(r j x + x j ) r 2j


x j ∈ 0(u) ,

where u ∈ P1+ (M) and r j → 0. The function u may even change for different j. Fact 1. The boundary ∂ has locally finite (n − 1)-Hausdorff measure (see [CKS], [Ca3] for details). 1 . Then it Fact 2. Let u j be a blow-up of u, and suppose that u j converges to u 0 in Cloc it is known that {u 0 = |∇u 0 | = 0} ⊃ lim{u j = |∇u j | = 0}

and interior({u 0 = |∇u 0 | = 0}) ⊂ lim{u j = |∇u j | = 0} (see, e.g., [Ca2], [KS], [CKS]); here lim denotes the limit set of all sequences {x j }, x j ∈ {u j = |∇u j | = 0}. Fact 3. Recall the definition of the class P1 (M). It is known (see [CKS]) that if u ∈ P1 (M) is ε-close to a half-space solution, ((x1 )+ )2 /2, say, then u ≡ 0 in B(0, 1/2) ∩ √ {x1 < −C ε} for some constant C > 0. Hence it is easy to deduce that if u ∈ PR+ (M) + (R > 4) and if u is ε-close in B R+ to ((x − 1)+ )2 /2, then u ≡ 0 in {x ∈ B R−2 : x1 < √ 1 − c ε} for small enough ε. Indeed, by the above statement, u ≡ 0 in the stripe √ {x = (x1 , X ) : x1 ∈ [1/2, 1 − C ε], |X | < R − 5/4}. Since u is subharmonic, it follows from the maximum principle that for any y = (1/2, Y ) (|Y | ≤ R − 7/4), either u ≡ 0 in B1/2−C √ε (y) or u(x 0 ) > 0 for some point x 0 ∈ B1/2−C √ε (y). But the latter is impossible, as follows from (3.1) and ε-closeness. Thus we obtain u ≡ 0 in a √ √ wider strip {x : C ε < x1 < 1 − C ε, |X | < R − 7/4}. By the same reasoning, √ u ≡ 0 in {x : 0 < x1 < C ε, |X | < R − 2}. Fact 4. According to [CKS, Th. III], there exists an ε0 (small enough) such that if u ∈ P1 (0, M) is ε0 -close to a half-space solution, ((x1 )+ )2 /2, say, then ∂ is the graph of a C 1 -function over 5 in Bt0 for a universal t0 > 0. Fact 5. The free boundary ∂ has zero Lebesgue measure (see [CKS, General remarks]). Fact 6. (Stability) Let u ∈ P1+ (0, M). Then from the nondegeneracy (3.1) we conclude that for any scaling u r of u with x 0 = 0, we have 0 ∈ 0(u r ). Moreover, the same holds for the limit function u 0 when it exists. Fact 7. Let u ∈ P1 (0, M), and let u j be a sequence of scalings of u. Then, for a subsequence, the second derivatives of u j converge to u 0 in L p ; that is, for some

REGULARITY PROPERTIES OF A FREE BOUNDARY

{ jm }, Dik u jm → Dik u 0

13

p

in L loc (Rn+ )-norm

for 1 < p < ∞ (see [CKS, General remarks (c)]). Fact 8. The continuation of a nonnegative subharmonic function v in  across ∂ by zero is subharmonic if v = 0 on ∂. 4. Proof of Theorem B In this section we prove Theorem B; that is, we classify all global solutions in the + (M). Obviously, the quadratic growth assumption in the definition of the class P∞ + class P∞ (M) is not redundant since u(x) = x12 /2 + x1 x2 x3 is a solution but not of quadratic growth. However, for our local class P1+ (0, M), we have that scaling of the + (0, M). members of this class remains (in the limit) in the class of global solutions P∞ + 2 It follows from Theorem A that for the class P∞ (M), the estimate |D u| ≤ C M in Rn+ is true. We leave it to the reader to verify this simple fact. The classification of global solutions in P∞ (0, M), that is, solutions in the entire space rather than in the upper half-space, was considered by Caffarelli, L. Karp, and Shahgholian in [CKS]. They showed that global solutions are either homogeneous polynomials of second degree or they are convex; that is, Dee u ≥ 0 for all directions e. This, in turn, means that the set 3(u) is convex. They also proved that if the set 3(u) is large enough at ∞, then the solution is a half-space solution; that is, u(x) = ((x1 )+ )2 /2 in some rotated coordinates. In our case, the situation is rather different. Since we already have the “forcing” set 5, it seems that this information is enough to enable us to prove the theorem under consideration. However, the proof requires some detailed analysis of the behavior of the free boundary far away. The proof is lengthy, and we have attempted to make it clear by adding some details that may not be obvious for nonexperts but are probably trivial to experts. Since the function vanishes on the set {x1 = 0} and since this set is large, we can use the monotonicity formula more effectively than for the interior case (see [CKS]). For e orthogonal to e1 , we first extend (De u)± as zero functions across 5 to Rn− . Then we notice that the scaled functions u r j (x) = u(r j x)/r 2j with r j → ∞ have uniformly bounded second derivatives. It follows from standard compactness arguments that a subsequence converges uniformly on compact subsets. Taking further a subsequence, 2, p 1,α
n we may also assume convergence of the sequence in Wloc ∩ Cloc (R+ ∪ 5) for any 1 < p < ∞ and 0 < α < 1 (see Fact 7). Indeed, we redefine r j to get the convergent sequence, and we denote by u ∞ the limit function for this sequence. Consider now the monotonicity formula for 0 < r <

14

SHAHGHOLIAN and URALTSEVA

r j to conclude that ϕ(r, De u) ≤ ϕ(r j , De u) ≤ lim ϕ(r j , De u) =: Ce , r j →∞

(4.1)

where e is orthogonal to e1 . Also, Ce exists by the monotonicity formula and the quadratic growth of u. Now Ce = lim ϕ(sr j , De u) = lim ϕ(s, De u r j ) = ϕ(s, De u ∞ ) r j →∞

r j →∞

for every s > 0; that is, ϕ(s, De u ∞ ) is constant for all s > 0. Since {De u ∞ = 0} ⊃ {x1 < 0}, Lemma 1.4 implies that either ϕ 0 (r ) > 0 for all r > 0 or ϕ(r ) ≡ 0. It thus follows that Ce = 0, and consequently, by (4.1), 0 ≤ ϕ(r, De u) ≤ Ce = 0. We thus conclude that De u ≥ 0 (or ≤ 0); both cases are treated similarly. We also observe that by the strong maximum principle De u > 0 in (connected components of)  or De u ≡ 0. If there is no direction e orthogonal to e1 such that De u 6≡ 0, then u depends only on the x1 -direction, and consequently it has the representation indicated in the theorem. So suppose, for some e (orthogonal to e1 ), De u > 0 in . By rotation (of 5), we may assume that e = e2 (the unit vector directed in the x2 -direction). Fix a point x 0 ∈ . Then in the x2 x3 -space (denoted T (e2 , e3 )) there must be a vector e for which De u(x 0 ) = 0. This is obvious; since D−e2 u < 0 and it is continuous in e, there must exist real numbers a, b such that De u(x 0 ) = 0 with e = ae2 + be3 . Now from the above we also have that De u does not change sign and it vanishes 0 at x . Therefore, by the maximum principle, it is identically zero in the connected component 0 of  containing x 0 . This implies that u restricted to 0 ∩ T (e1 , e2 , e3 ) is two-dimensional; here T (e1 , e2 , e3 ) means the x1 x2 x3 -space. Now repeating the same argument for other directions we conclude that u is two-dimensional in 0 . This can also be done for any connected component of . From the above reasoning it follows that we need to consider only the twodimensional problem. Now suppose that the upper half-space is free of boundary points, that is, that ∂ ⊂ 5. Then v = D2 u is harmonic in {x1 > 0} and it vanishes on 5. Therefore we can continue v harmonically into Rn . Since v has linear growth (by the assumption) and v(0) = 0, it follows by Liouville’s theorem that v = ax1 + bx2 . Now v = 0 on 5 implies that b = 0. Integration now gives u = ax1 x2 + g(x1 ). Next, 1u = 1 in the upper half-space gives g 00 (x1 ) = 1 there; that is, g(x1 ) = x12 /2 + αx1 + β. Since u(0) = 0, we must have β = 0. This shows that u(x) = x12 /2 + ax1 x2 + αx1 in the upper half-space. This proves the first part of the theorem.

REGULARITY PROPERTIES OF A FREE BOUNDARY

15

Next suppose that there exists x 1 ∈ ∂ with x11 > 0. Then, according to the above analysis and the fact that the free boundary has Lebesgue measure zero (see Fact 5 in Sec. 3), we conclude that 3(u) has nonvoid interior. Let us fix a ball B(x 0 , 2r ) in the interior of 3(u). Since D2 u ≥ 0, we must have u ≤ 0 on the half-cylinder generated by B(x 0 , 2r ) and directed towards the negative x2 -axis, that is, u ≤ 0 in  K (x 0 , 2r ) = (x1 , x2 − s); (x1 , x2 ) ∈ B(x 0 , 2r ) s ≥ 0 . From here one infers that for the smaller half-cylinder K (x 0 , r ) we have ∂ ∩ K (x 0 , r ) = ∅.

(4.2)

Indeed, if y ∈ K (x 0 , r ) ∩ ∂, then the strong maximum principle applied to u (a subharmonic function) in B(y, r ) gives that u(y) < 0 or it is identically zero. Since u(y) = 0, the only possibility is that u ≡ 0 in B(y, r ). Hence y 6∈ ∂(u). Next using (4.2) and the fact that B(x 0 , r ) ⊂ 3(u), we conclude that K (x 0 , r ) ⊂ 3(u). Hence we can translate u in the x2 -direction by considering u m (x) = u(x1 , x2 − m) in Rn+ . One first checks that |u m (x)| ≤ C(1 + |x|2 ) for all positive integers m.

(4.3)

This follows from the fact that for any x we evaluate u at x m = x − me2 and that the distance from x m to the set K (x 0 , r ) ⊂ 3 is less than or equal to |x − x 0 |. In other words,
dist x m , K (x 0 , r ) ≤ |x − x 0 |. Hence the quadratic growth (see Th. A) of u from the free boundary gives the desired bound. Since D2 u ≥ 0, the sequence {u m } is nonincreasing, and therefore by compactness it converges to a limit function u ∞ . Since u(x1 , x2 − m) ≥ −C(1 + x12 ) and since D2 u ≥ 0, there must be a limit for u at (x1 , −∞). In particular, u ∞ (x) = limm u(x1 , x2 − m), and it is independent of x2 . In other words, u ∞ is one-dimensional. It is now easy to conclude that u ∞ = ((x1 − a)+ )2 /2 for some constant a > 0. From here we see that u becomes nonnegative as x2 tends to −∞. Since D2 u ≥ 0, u ≥ 0 in the upper half-space. We now claim that u depends only on x1 . If this is true, then we are done using elementary calculus. First notice that u ≥ 0 implies that for any direction ν pointing inwards to Rn+ we have Dν u ≥ 0 on 5. (4.4) The objective is to prove that Dν u ≥ 0

in Rn+ for all ν : ν · e1 ≥ 0.

(4.5)

16

SHAHGHOLIAN and URALTSEVA

Obviously, (4.5) implies that u is one-dimensional. By (4.4) and the fact that Dν u = 0 on ∂ ∩ Rn+ , we may use the monotonicity formula for Dν u. Here is how. We first extend u across 5 by odd reflection to the largest set for which Dν u ≥ 0 below the plane 5. We also define Dν u to be zero in the rest of Rn− . By this definition, we have Dν u ≥ 0

in Rn− ,

(4.6)

and 1(Dν u)± ≥ 0. That 1(Dν u)− ≥ 0 follows from the fact that the support of (Dν u)− is in Rn+ . To deduce the subharmonicity of (Dν u)+ , one observes that Dν u ≥ 0 on 5, so that the odd reflection of u brings us the following information: (a1) D11 u(x) = 1 for x ∈ 5 \ 3(u) and x1 = 0+ , (a2) D11 u(x) = −1 for x ∈ 5 \ 3(u) and x1 = 0− , (b) Dνe u is continuous across 5 \ 3(u) for e ⊥ e1 . Consequently, (Dν u)+ is subharmonic in the open set {Dν u > 0}, where u is now the extended function. Since it is also continuously zero outside this set, it must be subharmonic in the whole space. Next we consider the monotonicity formula for Dν u over the whole space Rn . As in the previous situation for (4.1) and its succeeding argument (the reader should be aware of the fact that scalings of Dν u are uniformly Lip(Rn )), we have ϕ(s, Dν u ∞ ) = Cν for all s ≥ 0. We prove that Cν = 0. This then implies, as in the previous situation, that Dν u does not change sign, and hence it is nonnegative. Now suppose Cν 6= 0. Then according to the monotonicity formula, both of the sets Vs+ = {Dν u ∞ > 0} ∩ ∂ B(0, s),

Vs− = {Dν u ∞ < 0} ∩ ∂ B(0, s)

are half-spherical caps (up to (n − 1)-dimensional zero measure). This, however, is impossible due to the construction of the sets Vs± . Indeed, by (4.6), Dν u ≥ 0 in Rn− , and hence Dν u ∞ ≥ 0 in Rn− . Then the only possibility for Vs− to coincide with a halfspherical cap for any s > 0 is the condition Dν u ∞ ≤ 0 in Rn+ . This in turn implies that u ∞ ≤ 0 in Rn+ . In this case we have u ∞ ≡ 0 and again Cν = 0. This completes the proof. 5. Proof of Theorem C The proof of Theorem C now follows easily by using the characterization of global solutions. We first formulate several lemmas. LEMMA 5.1 Given ε > 0, there exists ρ = ρε such that if u ∈ P1+ (0, M), then for x 0 ∈ ∂(u) ∩ B + (0, ρε ) we have x 0 ∈ Bρε \ K ε , (5.1)

REGULARITY PROPERTIES OF A FREE BOUNDARY

where

17

q n o K ε := x : x1 > ε x22 + · · · + xn2 .

Proof The proof follows using a contradictory argument. So suppose that the conclusion of the lemma fails. Then there exist u j ∈ P1+ (0, M) and x j ∈ ∂(u j ) ∩ B1+ with |x j | & 0, such that (5.1) fails for x j . Then define u˜ j (x) = u j (d j x)/d 2j , where d j = |x j |. Now for each function u˜ j we have a point x˜ j = x j /d j ∈ ∂(u˜ j ) ∩ ∂ B1+ with q j j j x˜1 > ε (x˜2 )2 + · · · + (x˜n )2 . Next, by standard compactness and for a subsequence, u˜ j and x˜ j converge to u 0 and x 0 , respectively, with u 0 a global solution and x 0 ∈ K ε ∩ ∂ B1 . Moreover, by Fact 2, both x 0 and the origin are on the free boundary 0(u 0 ). Since u 0 is a global solution, this contradicts Theorem B. LEMMA 5.2 Given ε > 0, there exists ρ = ρε such that if u ∈ P1+ (M), x 0 ∈ 0(u) ∩ B + (0, 1/2), √ and x10 < ρε , then u ≡ 0 in B + (x 0 , 2x10 ) ∩ {x1 < x10 (1 − C ε)} and it is (ε(x10 )2 )close to a half-space solution ((x1 − x10 )+ )2 /2 in B + (x 0 , 3x10 ).

Proof In view of Fact 3, it suffices to prove (ε(x10 )2 )-closeness. So suppose that the second + statement in the lemma fails. Then there exist u j ∈ P1+ (M), x j ∈ ∂(u j ) ∩ B1/2 , such that j ((x1 − x1 )+ )2 sup u j (x) − > εd 2j , 2 + j B (x ,3d j ) j

where d j = x1 → 0. + Set v j (x) = u j (d j (x − e1 ) + x j )/(d j )2 . Then v j ∈ P1/2d (C Md −2 j ), e1 ∈ j ∂(v j ), and the following hold: sup |Dkl v j | ≤ C M,

B + (e1 ,r )

and

sup |v j | ≤ C Mr 2 ,

B + (e1 ,r )

∀r ≤

1 , 2d j

((x1 − 1)+ )2 sup v j − > ε. 2 B + (e1 ,3)

Now, for a subsequence and in an appropriate space, v j converges to a global + (C M) with e ∈ ∂(v ). But, on the other side, we have solution v0 ∈ P∞ 1 0 ((x1 − 1)+ )2 sup v0 − ≥ ε. 2 B + (e1 ,3)

18

SHAHGHOLIAN and URALTSEVA

This contradicts the second representation in Theorem B and completes the proof of the lemma. LEMMA 5.3 Given ε > 0, there exists ρε such that if u ∈ P1+ (0, M) and δρ (u) > ε for some ρ < ρε , then u > 0 in Bρ ∩ K ε , (5.2)

where K ε is defined in Lemma 5.1. Proof Suppose, towards a contradiction, that (5.2) fails, that is, that there exist ρ j & 0, u j ∈ P1+ (0, M), and x j ∈  ∩ Bρ j ∩ K ε such that u j (x j ) ≤ 0 and δρ j (u j ) > ε.

(5.3)

Now consider two different scalings of u j , u˜ ρ j =

u j (ρ j x) ρ 2j

,

u˜ r j =

u j (r j x) r 2j

,

where r j = |x j |. By (5.3) and the fact that u j (x j ) ≤ 0, one easily verifies that 1 2 1 a x1 , u˜ r j → P2 = x12 + x1 x2 , in Rn+ 2 2 2 for some constant a with |a| ≥ ε. Observe that we have used the fact that 0 ∈ 0(u), which in conjunction with Fact 6 gives 0 ∈ 0(Pi ), i = 1, 2. Moreover, we have r j ≤ ρ j . Now using the monotonicity formula, we show that this is not possible. Consider the usual odd reflection of u, so that any u ∈ P1+ (0, M) is defined in the whole unit ball. Now let 0 < β < (Cn ε4 )/4, and choose j large enough such that u˜ ρ j → P1 =

Cn ε4 ≤ Cn a 4 = ϕ(1, D2 P2 ) ≤ ϕ(1, D2 u˜ r j ) + β = ϕ(r j , D2 u j ) + β ≤ ϕ(ρ j , D2 u j ) + β = ϕ(1, D2 u˜ ρ j ) + β ≤ ϕ(1, D2 P1 ) + 2β Cn ε4 . (5.4) 2 We thus reach a contradiction. The reader should note that in (5.4) we have used the monotonicity formula at only one place, and while doing this we have fixed u j . = 2β ≤

5.4 Given ε > 0, there exists ρε > 0 such that if u ∈ P1+ (0, M) and δρ (u, 0) > ε for some ρ < ρε , then |De u(x)| < εx1 in Bρ+ (e ⊥ e1 ). LEMMA

REGULARITY PROPERTIES OF A FREE BOUNDARY

19

In Lemma 5.4 we assume that ε ≤ (8C1 M(n − 1))−1 , where C1 is as in Theorem A. Proof We argue by contradiction, using the structure of global solutions. First we prove that the statement holds outside a narrow region of the set 5. More precisely, we claim that + the estimate |De u(x)| < εx1 /2 holds in B2ρ ∩ {x1 > ερ} for small enough ρ. Indeed, + if this fails, then there exist sequences u j ∈ P1+ (0, M), ρ j → 0, x j ∈ B2ρ ∩ {x1 > j ερ j }, and ν j ⊥ e1 (vectors) such that j

δρ j (u j , 0) > ε,

|Dν j u j |(x j ) ≥

εx1 . 2

By Lemma 5.3, u j > 0 in B2ρ j ∩ K ε/2 for 2ρ j < ρε/2 . Scale u j by ρ j at the origin; that is, define u j (ρ j x) u˜ j (x) = . ρ 2j Then

j

ε x˜1 , 2 where x˜ j = x j /ρ j ∈ B2+ ∩ {x1 > ε}. As j tends to ∞, we can extract convergent + (0, M), subsequences of u˜ j , x˜ j , and ν j with the corresponding limit values u˜ 0 ∈ P∞ + x˜ 0 ∈ B2 ∩ {x1 ≥ ε}, and ν0 ⊥ e1 such that u˜ j > 0

in K ε/2 ∩ B2+ ,

|Dν j u˜ j |(x˜ j ) ≥

u˜ 0 ≥ 0 in K ε/2 ∩ B2+ ,

|Dν0 u˜ 0 |(x˜ 0 ) ≥

ε x˜10 . 2

+ (0, M). Now the classification of global solutions (see Th. B) Moreover, u˜ 0 ∈ P∞ implies that in Rn+ , 1 u˜ 0 (x) = x12 + ax1 x2 , 2 after suitable rotation. Hence D2 u˜ 0 = ax1 . Now the nonnegativity of u˜ 0 in K ε/2 ∩ B2 in conjunction with freshman calculus implies that 2|a| ≤ ε/2. Thus x˜ 0 ∈ B2+ ∩{x1 ≥ ε} implies that

ε x˜10 ≤ 2|D2 u˜ 0 (x˜ 0 )| = 2|a|x˜10 ≤

ε x˜10 , 2

x˜10 > 0,

and we reach a contradiction. Now, to prove the lemma, we need to estimate |De u| in the narrow region {x1 ≤ ερ} ∩ Bρ . We use a barrier argument. Let v ± (x) = (De u)± , and let w(x) = εx1 −

x12 ε + 4C1 M |X − Y |2 , 2ρ ρ

20

SHAHGHOLIAN and URALTSEVA

where Y ∈ 5 ∩ Bρ and X = (0, x2 , . . . , xn ). Now v ± are subharmonic, w is superharmonic, and w ≥ v ± on the boundary of the cylinder Q(Y ) = {x : 0 < x1 ≤ ερ; |X − Y | < ρ/2}. Now applying the comparison principle, we obtain v ± ≤ w in Q(Y ). In particular, for X = Y we have the estimate v ± (x1 , Y ) ≤ εx1 for 0 < x1 < ερ. Now varying Y , we come to the desired estimate in the half-ball B + (0, ρ). Recall the definition of singular contact points Su . LEMMA 5.5 Let u ∈ P1+ (0, M), and suppose that x = 0 is a singular contact point. Then all blow-ups of u at the origin have the same representation as in (1.2), with a fixed a.

Proof First let us extend u across 5 by odd reflection. Let r j & 0 be such that u r j = u(r j x)/r 2j converges, for a subsequence, and in appropriate spaces, to a global solution 1 1X u 1 = x t Ax = ai j xi x j , 2 2 where A is a symmetric matrix with entries ai j . More exactly, ai j = 0 for min(i, j) > 1, and a11 = 1. Now let t j & 0 be another arbitrary sequence, and define accordingly u t j . Then a similar argument gives a limiting polynomial u2 =

1X 1 t x Bx = bi j xi x j . 2 2

Here B = (bi j ) is a symmetric matrix, and as above bi j = 0 for min(i, j) > 1, and b11 = 1. Now for a directional vector e ⊥ e1 , consider the monotonicity formula for De u. Then ϕ(r, De u) is a monotone nondecreasing function of r . Since ϕ is monotone (see Lem. 1.4), the limit, as r tends to zero, exists and lim ϕ(r, De u) = lim ϕ(1, De u r ) = Ca 4 .

r →0

r →0

Observe that either a 6= 0 or that the blow-up u 0 satisfies De u 0 ≡ 0. Replacing r by r j and then by t j , we obtain ϕ(1, De u 1 ) = ϕ(1, De u 2 ).

(5.5)

Inserting the polynomial representations of u 1 and u 2 in (5.5), we obtain for all vectors e ⊥ e1 , kAek = kBek, (5.6)

REGULARITY PROPERTIES OF A FREE BOUNDARY

21

where k · k denotes the usual vector norm. From here we show that A = B. First notice that (5.6) implies that the kernels of A and B are the same. In particular, we may rotate the plane 5 in order to obtain a11 = b11 = 1, ai j = bi j = 0 for i + j ≥ 4. In other words, we have the representation u1 =

1 2 x + ax1 x2 , 2 1

u2 =

1 2 x + bx1 x2 . 2 1

Next, using (5.6) for a different choice of directions e, we have |a| = |b|. If a = 0, then we are done. Otherwise, to see that a = b, it suffices to show that {u 1 < 0} ≡ {u 2 < 0}. We consider a point x 0 ∈ ∂ B1+ such that u 1 (x 0 ) < 0 but u 2 (x 0 ) > 0, and we show that this results in a contradiction. Define the segment l0 = {t x 0 : 0 ≤ t ≤ 1}. Then we claim that there exists r0 such that u does not change sign on l0 ∩ Br0 . If this holds, then we are done. So suppose that this fails. Then there exists ρ j → 0 such that u(ρ j x 0 ) = 0. Scale u by ρ j ; that is, define u ρ j (x) = u(ρ j x)/ρ 2j . Applying the monotonicity formula (after odd reflection of u) as above, we end up with a limit function (global solution) that has the representation u 3 = x12 /2 + cx1 x2 . Moreover, the above analysis gives |c| = |a|. Hence c = a or c = b; that is, u 3 = u 1 or u 3 = u 2 . In particular, u 3 (x 0 ) 6= 0. Using the information that u ρ j (x 0 ) = 0, we also conclude that u 3 (x 0 ) = 0. This is a contradiction. For the next lemma we need a definition of lower-dimensional cones. We set K ε0 (u) := {x ∈ 5 : |x · e A | > ε|x|}, where e A ∈ 5, e A ⊥ Ker(A), ke A k = 1. Here A is the unique matrix corresponding to the blow-up of u at the origin. Observe that the uniqueness of A, and thus the uniqueness of e A , up to the factor −1, is guaranteed by Lemma 5.5. 5.6 Given ε > 0, there exists rε = rε (α) such that if 0 ∈ Su (α), then LEMMA

0(u) ∩ Brε ∩ K ε0 (u) = ∅. The proof of Lemma 5.6 uses ideas similar to those in Lemma 5.4. Proof If the conclusion fails, then there exist u j ∈ P1+ (0, M) with 0 ∈ Su j (a j ) (where |a j | ≥ α) and x j ∈ 0(u j ) ∩ K ε0 (u j ) with r j := |x j | → 0. Let A j be the unique matrix corresponding to the blow-up of u j (see Lem. 5.5). To simplify the analysis, we may consider a rotated system for each function such that the blow-up of u j at the origin is the polynomial P j = x12 /2 + a j x1 x2 .

22

SHAHGHOLIAN and URALTSEVA

Define as usual the rescaled function u˜ j (x) = u j (r j x)/r 2j . Then x˜ j = x j /r j ∈ 0(u˜ j ) ∩ K ε (u˜ j ). Also, |x˜ j | = 1. Extracting a subsequence of all the ingredients, we have a limit function u 0 , a limit point x 0 ∈ 0(u 0 ) ∩ K ε0 (u 0 ), and finally a limit value a0 with |a0 | ≥ a. The problem is that the matrix A associated to the blow-up u 0 does not need to have the same fixed kernel as all the A j , and we have no contradiction. However, the monotonicity formula helps us to fix this small detail. First notice that 0 ∈ Su j (a j ) implies that C(a j ν j · e2 )4 = lim ϕ(r, Dν j u j ), r →0

where ν j is the directional vector 0x˜ j . Now applying the monotonicity formula once again, we have C(αε)4 ≤ C(a j ν j · e2 )4 ≤ ϕ(r j , Dν j u j ) = ϕ(1, Dν j u˜ j )

(5.7)

for all j = 1, 2, . . . . Letting j → ∞ in (5.7), we obtain 0 < C(αε)4 ≤ ϕ(1, Dν0 u 0 ),

(5.8)

where ν0 = 0x 0 . Now by Theorem B, u 0 is a two-degree homogeneous polynomial of one or two dimensions. Since x 0 ∈ 0(u 0 ), we conclude, by homogeneity, that the whole line through x 0 and the origin is on the free boundary 0(u 0 ). In particular, u 0 is independent of the direction ν0 , and hence Dν0 u 0 ≡ 0. This contradicts (5.8). Hence we have reached a contradiction, and the lemma is proved. 0

Let A x be the matrix corresponding to the blow-up of u at x 0 (see the proof of Lem. 5.5). In the next lemma, using ideas similar to those in the proof of Lemma 0 5.6, we can prove that the kernel of A x is continuous in x 0 for x 0 ∈ Su (a), where the continuity depends on the constant a. For this purpose we need a definition of distance of the matrices. For two (n × n)-matrices A1 and A2 , we define dist(A1 , A2 ) as the angle between the corresponding vectors e A1 and e A2 (see the definitions preceding Lem. 5.6). LEMMA 5.7 Given ε > 0, there exists rε = rε (a) > 0 such that if x 0 , x 1 ∈ Su (a) and |x 0 − x 1 | < 0 1 rε , then dist(A x , A x ) < ε.

The proof follows by Lemma 5.6. We leave the details to the reader. Proof of Theorem C For each case we define a modulus of continuity σ (r ) and a radius r0 ; then we take the minimum of all of these as the one given in the statement of the theorem.

REGULARITY PROPERTIES OF A FREE BOUNDARY

23

(C1) For the first statement in the theorem we consider the modulus of continuity σ (t) given by the inverse of the relation ε → ρε in Lemma 5.1. Now let r0 = ρ{ε=1} . (C2) For any x 0 ∈ ∂ with x10 > 0 and small enough, we can use Lemma 5.2 to conclude that in B(x 0 , 3x10 ) the function u is (ε(x10 )2 )-close to the half-space solution ((x1 −x10 )+ )2 /2, and there are no contact points there. Hence [CKS, Lem. 6.2] applies, and we deduce that D1 u − u ≥ 0 and u ≥ 0 in B(x 0 , 3x10 /4). In particular, by the maximum principle, D1 u > 0 in  ∩ B(x 0 , 3x10 /4). Thus we can apply the result of [AC] (cf. also [Ca3, Th. 7]) to conclude that ∂∩ B(x 0 , x10 /4) is a C 1,α -graph over 5. Now let us denote by σ (r ) the modulus of continuity for the inward normal vector to the free boundary that appears in the result of [AC]. Then the modulus of continuity of the inward normal vector to ∂ ∩ B(x 0 , x10 /4) is given by σ (r/x10 ), which obviously deteriorates as x 0 comes closer to the plane 5. The latter depends on the scaling upon application of [AC]. Finally, to complete the proof of this part of the theorem, we need to show that the (inward) normal vector to ∂ is uniformly continuous. Therefore, by Lemma 5.2, [CKS, Lem. 6.2], and [Ca3, Th. 7], for given ε > 0 we can choose rε small enough + such that for x ∈ ∂ ∩ B1/2 with x1 < rε we have Angle(nx , e1 ) <

ε , 2

(5.9)

where nx is the inward normal vector to ∂ at x. + Then for y 1 , y 2 ∈ ∂ ∩ B1/2 we consider three cases: 1 2 (I) y , y ∈ {x1 < rε /2}, (II) y 1 , y 2 ∈ {x1 ≥ rε /2}, (III) y 1 ∈ {x1 < rε /2}, y 2 ∈ {x1 ≥ rε /2}. In case (I), we may use (5.9) to deduce Angle(n y 1 , n y 2 ) < ε. In case (II), we take |y 1 − y 2 | ≤ sε with sε such that σ (sε /rε ) < ε. In case (III), by restricting the distance |y 1 − y 2 | ≤ rε /2 and by using (5.9), we conclude that Angle(n y 1 , n y 2 ) < ε. Combining all three cases, we may define our modulus of continuity as the inverse relation to n r o ε ε → min sε , . 2 (C3) The third statement in Theorem C is now an easy consequence of Lemmas 5.6 and 5.7. Since the singular points appear only when the set 3r (u) ∩ 5 lies in a narrow cusplike region (see Lem. 5.6), one may argue in a classical way to deduce the statement in part (C3). Obviously, the continuity of the kernel of A x in Lemma 5.7 plays an essential role. For the reader’s convenience, we carry out some details. Let u 0 be a blow-up of u at x. As a simple application of the classification in Theorem B, Lemma 5.5, and (C2), we obtain that all blow-ups u 0 of u at a fixed point

24

SHAHGHOLIAN and URALTSEVA

(x here) are the same, and, moreover, the set 3(u 0 , x) is a linear space of dimension (n − 2). This is actually the tangent space of the contact points at x. According to Lemma 5.6, the singular points of Su (a) have the property that
Angle x x k , 3(u 0 , x) → 0 (5.10) as x k ∈ 3(u 0 , x), and x k → x. The notation in (5.10) stands for the angle between the vector x x k and the linear subspace 3(u 0 , x). Let the inverse relation ε → rε in Lemma 5.6 be denoted by σ (r ). Then by Lemma 5.6, the angle in (5.10) tends to zero with a uniform speed of σ (r ), where r = |x − x k |. This (obviously) proves that the set of singular contact points has the property stated in the theorem. At this step, Lemma 5.7 is crucial. One just uses the classical Whitney-type extension to show this (see more details in [CSh]). j (C4) Let x j → 0 nontangentially; that is, let x1 ≥ C|x j | for some C > 0. Define u j (x) =

u(|x j |x) , |x j |2

x˜ j =

xj ∈ ∂ B1 . |x j |

Obviously, j

x˜1 ≥ C. Since, by Lemma 5.1, B(x˜ j , C/2) ⊂ (u˜ j ), we have u j → u 0 (some limit function) in C 2 (B(x˜ j , C/4)). In particular, D 2 u j (x˜ j ) = D 2 u(x j ) → D 2 u 0 (x 0 ). Now, by Lemma 5.5, all blow-ups at the origin have the same representation. (They are the same function.) Hence u 0 = (x t Ax)/2 for some symmetric matrix A with entries ai j as in Lemma 5.5. Moreover, A is independent of the blow-up, that is, independent of the choice of x j . This gives  x t Ax  D 2 u j (x j ) → D 2 = fixed. 2 This proves the claim. (C5) The fifth statement uses Lemma 5.4. Here we define the modulus of continuity σ (τ ) to be the one given by the inverse relation of ε → ρε . We remark that the thickness condition of the statement in this case is absolutely necessary since otherwise we have the counterexamples of the type D21 P2 ≡ a. (C6) The last statement for n = 2 is now obvious. We treat the case of n > 2. First notice that, by the techniques in the proof of Lemma 5.3, one may conclude that the assumption limr →0 δr (u, 0) > 0 gives that any blow-up u 0 at the origin is one-dimensional; that is, u 0 = x12 /2. Let us now consider an arbitrary sequence x j → 0 and prove that Dνe u(x j ) → 0 for e ⊥ e1 and ν arbitrary. There are two separate cases to be considered:

REGULARITY PROPERTIES OF A FREE BOUNDARY

25

d(x j , 0(u)) = o(d(x j , 00 )), where d(·) denotes the distance function; d(x j , 0(u)) ≥ c0 (d(x j , 00 )) for some c0 > 0. In case (a), the contact points are far away from the point x j relative to the distance from x j to the free boundary points in the upper ball. Therefore, in view of (C2), scaling with respect to the nearest point y j ∈ 0(u) eventually leads to a problem, as in the interior case, and the result follows by [CKS]. In case (b), let y j ∈ 00 (u) be any nearest point to x j , and set r j = |x j − y j |, j z = (x j − y j )/r j . Now scale u as follows:

(a) (b)

u j (x) =

u(r j x + y j ) r 2j

.

Then Bc+0 (z j ) ⊂ (u j ) and by elliptic theory we have Dνe u(z j ) → Dνe u 0 (z 0 ), where z 0 = lim z j and where u 0 is the limit of u j (maybe for a subsequence). By definition, |z j | = 1, and hence |z 0 | = 1. On the other hand, using (C5), we get sup B1+

|De u j (x)| |De u(x)| |De u(x)| = sup ≤ sup ≤ σ (r j + 2|y j |) → 0 x1 x x + + 1 1 Br (y j ) B r j +|y j |

j

as j → ∞. Hence sup B + |De u 0 (x)| = 0 for any e ⊥ e1 . Thus, for any e ⊥ e1 and 1

arbitrary direction ν, we have Deν u 0 ≡ 0 in B1+ . Therefore Deν u(x j ) = Deν u j (z j ) → 0

as j → ∞.

6. Proof of Theorem D The proof of Theorem D is much the same as that of the case of the half-ball. The main three steps should be followed as in the half-ball situation. In other words, one first proves the C 1,1 -regularity (see Th. A). Since G is uniformly C 3 (observe that we may even change G, that is, consider a whole class of C 3 -surfaces), any blow-up of any sequence of functions in the class P1+ (0, M) results in a global solution in the upper half-space Rn+ . We thus have global solutions as before in Rn+ , after suitable rotation and translation. Now the classification of such solutions is done by Theorem B. The third part of the proof, that is, Theorem C type analysis, is done in a similar way since it is a local analysis and since blow-up always gives a global solution in the upper half-space. Despite the simplicity of the procedure described above, there is a “small” problem. The odd reflection, used in the case of 5 (a plane), is not possible in the general case of nonplane surfaces. Also, the monotonicity lemma is not applicable in this case since the tangential derivative De u (here e is tangential direction at the origin) is not necessarily zero on the rest of G.

26

SHAHGHOLIAN and URALTSEVA

Our main concern, in this section, is the technical difficulties discussed above. We discovered that this seems to be a nontrivial task. So let us formulate the problem more rigorously. First let us, in accordance with Definition 1.1, define the class Pr+ (0, M, M1 ), which corresponds to all functions having the properties in Definition 1.1 with B + replaced by G + in Theorem D, and with 5 replaced by G. Here M1 corresponds to the C 3 -norm of G. Suppose also that the normal vector to G at the origin is parallel to the x1 -axis and that G + lies above G (see Fig. 2 on p. 8). Definition 6.1 We say a (nonzero) function u belongs to the class P1+ (M, M1 ) if (for some ) u satisfies (in the sense of distributions) the following: (1) 1u = χ in G + for some  ⊂ G + , (2) u = |∇u| = 0 in G + \ , (3) kuk∞,G + ≤ M, (4) u = 0 on G ∩ B1 (0). Our strategy is very simple. We make the estimates ϕ(r, De u, x 0 ) ≤ C

for x 0 ∈ 0(u)

with Br (x 0 ) ∩ G 6= ∅. Then for all other (small) r we can use the monotonicity formula since we now have a solution in the entire ball Br (x 0 ). For a fixed small value r0 = r0 (M1 ) and a suitable rotation, there is a C 3 representation of the surface G in the following manner: G = {x ∈ Br0 : x1 = ψ(X )},

ψ(X 0 ) = 0, |∇ψ(X 0 )| = 0,

where X = (x2 , . . . , xn ) and ψ has C 3 -norm M1 , say. Also, the set G + ∩ Br0 (x 0 ) lies above the graph x1 = ψ(X ). For simplicity we assume X 0 = 0. LEMMA 6.2 Let e1 be the normal vector to G at the origin, and let e ⊥ e1 . Then we have the following estimate:

|De u(x) + D1 u(x)De ψ(X )| ≤ C|x1 − ψ(X )|

for x ∈ G + ∩ Br0 /2 .

Here C = C(n, M, M1 ). Proof First, as suggested by standard techniques in partial differential equations, one transfers the problem to the half-ball by the following change of coordinates: y1 = x1 − ψ(X ),

yi = xi

(i = 2, . . . , n),

(6.1)

REGULARITY PROPERTIES OF A FREE BOUNDARY

27

so that U (y) = u(x). Let y 0 and Q r+ (y 0 ) be the images of x 0 and Br (x 0 ) ∩ G + , respectively, under the mapping (6.1). We also define Q r+ = Q r+ (0). Rewrite the Laplacian in these new coordinates. This gives n X

ai j U yi y j + b1 U y1 = χ(U )

in Q r+0 ,

U (0, Y ) = |De U (0, Y )| = 0, (6.2)

i, j=1

where e is orthogonal to e1 (the x1 axis). Also, the coefficients ai j are represented by   (1 + |∇ψ|2 ) −∇ψ ai j = , b1 = −10 ψ, (6.3) −∇ψ I where 10 is the (n − 1)-dimensional Laplacian. Differentiating (6.2) with respect to yk (k > 1, i.e., tangential directions), we obtain n X

ai j U yk yi y j +

i, j=1

∂ai j ∂b1 U y y + b1 U y1 yk + Uy = 0 ∂ yk i j ∂ yk 1

in (U );

here (U ) is the transferred domain {y1 > 0}. Thus the function v = U yk , k = 2, . . . , n, satisfies the equation of the form in (U ) ∩ Q r+0 ,

(6.4)

on {y1 = 0} ∩ Q r+0 ,

(6.5)

Lv = g with boundary condition v=0

where Lv = ai j v yi y j and the L p -norms at the right-hand side of (6.4) can be estimated for p < ∞ as the following: (6.6)

kgk p,(U )∩Qr+ ≤ C p M 0

with a constant C p depending only on p and G. Now we are ready to apply [LU, Th. 2.1], which gives the estimate
v(y) ≤ C 1 + kgk p,Qr+ ∩{v>0} , 0 y1

y ∈ Q r+0 /2 ,

(6.7)

for functions v ∈ C(Q r+0 ) ∩ C 2 (Q r+0 ∩ {v > 0}) satisfying (6.5) and the inequality −Lv ≤ g

in Q r+0 ∩ {v > 0}.

(6.8)

Strictly speaking, [LU, Th. 2.1] is formulated for functions v ∈ C(Q r+0 ) ∩ C 2 (Q r+0 ) satisfying −Lv ≤ g in Q r+0 . But, in fact, the proof in [LU] implies the corresponding

28

SHAHGHOLIAN and URALTSEVA

information about v not for the whole Q r+0 but only for the part of it where v is positive. It follows from (6.4) – (6.7) that |U yk | ≤ C y1

in Q r+0 /2

(6.9)

for k = 2, . . . , n. Coming back to x-variables, we arrive at the statement of the lemma. The next lemma helps us to estimate the second derivatives. LEMMA 6.3 + Let u be in the class P1+ (M, M1 ). Suppose also that x 0 ∈ 0(u) ∩ B1/4 and Br (x 0 ) ∩ G 6= ∅. Then Z |D 2 u(x)|2 d x ≤ C0r 2 , (6.10) 0 |n−2 0 |x − x Br (x )∩V

where V = {x ∈ G + : ψ(X ) < x1 < ψ(X ) + r0 , |X | ≤ r0 } and C0 depends on P M, M1 and the dimension n. Here |D 2 u| = i j |Di j u|2 . Proof We start by rewriting the equation 1u = χ =: f in its weak form: Z Z − ∇u · ∇η d x = f η d x, ∀η ∈ C0∞ (V ). V

V

Now changing the variables (as in (6.1)) in the above integrals and using the same notation for η in the new variables, we arrive at Z Z − Ai j U y j · η yi dy = f˜η J dy, ∀η ∈ C0∞ (Q r+0 ), (6.11) Q r+0

Q r+0

where f˜ is the transferred f ,  ∂x  i , J = det ∂yj

Ai j = ai j J.

Now if in (6.11) we replace η by η yk (k = 2, . . . , n) and integrate by parts, we arrive at the following relation for the function v = U yk : Z Z h i ∂(J ai j ) Ai j v y j η yi dy = f˜ J δik − U y j η yi dy. (6.12) ∂ yk Q r+ Q r+ 0

0

It is clear that (6.12) holds for any η ∈ W 1,2 (Q r0 ). Now let ξ = ξ(|x − x 0 |) be a cutting function for a ball B2r (x 0 ), ξ = 1 in Br (x 0 ), r ≤ r0 /2. We keep the same notation for ξ in the y-variables. In (6.12), insert η = v9 N ξ 2 , where 9 N = min(9, N )

REGULARITY PROPERTIES OF A FREE BOUNDARY

29

and 9 is the fundamental solution for the divergence operator (∂/∂ yi )(ai j J (∂/∂ y j )) with pole at y 0 = y 0 (x 0 ). Here N is a large constant that tends to ∞. As a result of this, we obtain Z (ai j v yi )(v y j 9 N ξ 2 + 29 N vξ ξ y j )J dy + I Z h i  ∂(J ai j ) = J f˜δik − U y j v yi 9ξ 2 + v(9 N ) yi ξ 2 + 2v9 N ξ ξ yi dy, (6.13) ∂ yk where Z

J ai j (v ) yi 9 y j ξ dy =

Z

v J ai j (9 N ) y j ξ ξ yi dy +

Z

2

2I = {9
Z −2 Z ≥ −2

2

2

{9
{9=N }

v 2 J ai j (9 N ) y j ξ ξ yi dy,

J ai j (9 N ) y j


yi

v 2 ξ 2 dy

v 2 J ai j (9 N ) y j ξ 2 cos(γ , y j ) dy (6.14)

where γ is the unit normal vector to {9 = N } directed towards {9 > N }. Letting N → ∞ in (6.14) and using (6.13), (6.14), we have Z Z h 2 J ai j v yi v y j 9ξ dy ≤ − 2J ai j v yi v9ξ ξ y j − v 2 J ai j 9 y j ξ ξ yi i X +C |v yi 9ξ 2 + v9 yi ξ 2 + 2v9ξ ξ yi | dy. i

Now the assumption that Br (x 0 ) intersects G implies that y10 < Cr . Thus, by (6.9), we have |v(y)| ≤ Cr . Therefore, in the standard way, we can obtain Z |v yi |2 ξ 2 d x ≤ C0r 2 (i = 1, . . . , n) 0 n−2 Q r+ (y 0 ) |y − y | where v = u yk with k = 2, . . . , n. This estimate, in conjunction with (6.2) and (6.3), implies Z |U yi y j |2 ξ 2 d x ≤ C0r 2 (i, j = 1, . . . , n). 0 n−2 Q r+ (y 0 ) |y − y | Finally, changing the coordinates to the x-variables, we have the estimate (6.10). The proof is completed. Now we start the process of scaling and applying the monotonicity lemma as in [CKS, Lem. 3.2]. Recall the definition of S j (u) from Section 2. Then we claim that for u ∈ P1+ (M, M1 ) there exists a constant C0 = C0 (M, M1 , n) such that n S (z, u) o j S j+1 (z, u) ≤ max , C0 M2−2 j for all j, (6.15) 4

30

SHAHGHOLIAN and URALTSEVA

where z ∈ 0(u) ∩ V . Here V is as in Lemma 6.3. Now if (6.15) fails, then there exist sequences u j ∈ P1+ (M, M1 ), z j ∈ 0(u j ), and k j (→ ∞) such that n Sk (z j , u j ) o j Sk j +1 (z j , u j ) ≥ max , j M2−2k j . 4

(6.16)

Two situations may arise: (a) B2−k j (z j ) ⊂ G + , at least for some subsequence of k 0j of k j ; (b) B2−k j (z j ) ∩ G 6= ∅ for all k j with sufficiently large j. For case (a), we are just in the situation considered in [CKS, Lem. 3.2]. The only difference is that here we cannot apply the monotonicity formula to ϕ(r, De u j , z j ) for any r ≤ 1/2 because u j may not be defined in the whole ball Br (z j ). Let R j be the maximum radius of Br (z j ) for which Br (z j ) ⊂ G + . By Lemma 6.3, we have ϕ(R j , De u j , z j ) ≤ C for any direction e in Rn , and by the monotonicity formula, we 0 have ϕ(2−k j , De u j , z j ) ≤ C. The problem has now turned into an interior case, and we can use the proof in [CKS, Lem. 3.2]. For case (b), we need a more detailed analysis. Now one defines u˜ j (x) =

u j (2−k j x + z j ) Sk j +1 (z j , u j )

in the corresponding domain. More exactly, if j

ψ j (X ) :=

ψ(2−k j X + Z j ) − z 1 2−k j

,

G +, j := {x ∈ B1 (0) : x1 > ψ j (X )},

and G j := {x ∈ B1 (0) : x1 = ψ j (X )}, then by (6.16) we have sup G +, j ∩B1/2

|u˜ j | = 1,

sup |u˜ j | ≤ 4, G +, j

|1u˜ j | =

2−2k j 1 χ ≤ ( u ˜ ) j j Sk j +1 (z , u j ) jM

in G +, j . Now coming back to the sequence z j , we see that z j → z 0 ∈ G (for some z 0 ); by translation, we may assume z 0 = 0, ψ(0) = |∇ψ(0)| = 0. Consequently, ψ j (0) ≤ 0, and |∇ψ j (X )| = |∇ψ(2−k j X + Z j )| = O(2−k j + |Z j |) → 0 as j → ∞. The derivatives of ψ j , of order two and three, vanish in the limit as well. In other words, ψ j converges in C 3 to a constant c ∈ [−1, 0]. Now let Vc = {x ∈ B1 (0) : x1 > c}. If c < 0, then the origin is an interior point of G + and the proof in this case is similar to that of case (a). So suppose c = 0, so that the

REGULARITY PROPERTIES OF A FREE BOUNDARY

31

origin is a boundary point of Vc . Now using Lemma 6.2, we have, for e ⊥ e1 and x ∈ B2−k j (z j ) ∩ G + , the estimate |De u j (x) + D1 u j (x)De ψ(X )| ≤ C|x1 − ψ(X )| ≤ 2C2−k j . For the rescaled function u˜ j , we can use (6.16) and Lemma 6.2 to arrive at |De u˜ j (x) + D1 u˜ j (x)De ψ j (X )|   2−k j De u j (2−k j x + z j ) + D1 u j (2−k j x + z j )De ψ(2−k j X + Z j ) = j Sk j +1 (z , u j )  2C2−2k j  2C ≤ ≤ → 0, (6.17) Sk j +1 (z j , u j ) jM as j tends to ∞, and x ∈ B1 (0, ) ∩ G +, j . It is better now to transfer the problem to a fixed domain by changing variables as + in (6.1) with ψ j instead of ψ. Denote by z˜ j and Q r, j the images under this mapping j +, j ˜ of z and Br (0) ∩ G , respectively. Also, let U j (y) = u˜ j (x) = u˜ j (y1 + ψ j (Y ), Y ). + + For large j, one has Q + 1/2, j ⊂ B3/4 ⊂ Q 1, j and

sup |U˜ j | = 1, Q+ 1/2, j

sup |U˜ j | ≤ 4, C , j

(6.19)

on 5 ∩ B3/4 ,

(6.20)

kL ( j) U˜ j k∞,B + ≤ 3/4

U˜ j = 0 U˜ j (˜z ) = |∇ U˜ j (˜z )| = 0 j

(6.18)

+ B˜ 3/4

j

and

j

z˜ → 0

as j → ∞.

(6.21)

In (6.19) the coefficients of the operator ( j)

L ( j) = akl

∂2 ∂ − b( j) ∂ yk ∂ yl ∂ y1

are defined by relations similar to (6.3) with ψ j instead of ψ. It is easy to deduce from + (6.18) – (6.20) that the L p (B5/8 )-norms of the second derivatives of U˜ j are bounded uniformly with respect to j for any p < ∞. In particular, for some subsequence, + U˜ j → U0 weakly in W 2, p (B5/8 ) where the limit function U0 satisfies 1U0 = 0

+ in B5/8 ,

U0 = 0

+ on 5 ∩ B5/8 ,

and sup |U | = 1, + B1/2

U (0) = |∇U (0)| = 0.

(6.22)

32

SHAHGHOLIAN and URALTSEVA

Observe that the left-hand side in (6.17) coincides with |De U˜ j (y)|. Hence we get + De U 0 ≡ 0 in B1/2 for any e ⊥ e1 . Therefore U0 = U0 (y1 ) = by1 + c = 0. This contradicts the first statement in (6.22). Thus we have proved (6.15). From the above analysis it follows that we have an estimate similar to that in (2.1). Now the C 1,1 -regularity follows as in the proof of Theorem A. Now to prove Theorem C, we argue as in Section 5, without any essential changes. Indeed, Lemmas 5.1 and 5.2 go through without any changes. One only needs to verify that the limit of scalings of G is the plane 5. This, however, is elementary, and the reader may verify it easily. For the proof of Lemma 5.3, we used the monotonicity formula and the odd reflection of u. This, however, is not applicable in our case since the surface G is not a plane. Also, the function De u, where e is the orthogonal direction to e1 , is not necessarily zero on the rest of G. This is needed in order to apply the monotonicity formula in (5.4). Now the above reasoning calls for a new type of argument in the general case of Theorem D. We do this in detail. We first define a new function v(x) = D1 u Dk ψ + Dk u

in G + ,

where k = 2, . . . , n. Recall that the graph G has the normal e1 at the origin 0 ∈ G. The function v thus defined is zero on G ∩ Br0 . Now let B(x 1 , r1 ) be the ball touching G at the origin, and suppose that it is located outside G + ; that is, B(x 1 , r1 )∩ G + = ∅, and 0 ∈ ∂ B(x 1 , r1 ). Next consider an “odd” inversion of the function v with respect to ∂ B(x 1 , r1 ). We call this function v also. In other words, we define v(x) = −

r1n−2 v(r12 (x − x 1 )/|x − x 1 |2 ) |x − x 1 |n−2

for x ∈ B(x 1 , r1 ) ∩ G˜+ ,

where G˜+ is the inversion of G + in the same ball. Now let r2 ≤ r1 /2. Then v is defined in B(0, r2 ) \ (G + ∪ G˜ + ). Extend v as a zero function to the rest of B(0, r2 ). Now a lengthy but elementary calculation reveals that 1v ± ≥ −C in B(0, r2 ). It is also not hard to verify that, as we scale v, the set where v is zero converges to 5, the (n − 1)-dimensional plane, and that the blow-up of v converges to a global solution in the sense of Theorem B. At this point one realizes, probably with some effort, that all analysis involving the monotonicity formula may be carried out by using the monotonicity formula of [CJK, Lem. 1.5]. We leave the obvious details to the reader. Acknowledgment. This work was done while N. Uraltseva was visiting the MittagLeffler Institute, Djursholm, Sweden.

REGULARITY PROPERTIES OF A FREE BOUNDARY

33

References [ACF]

[AU]

[AC]

[ACS]

[BCN]

[BL]

[BLN]

[Ca1] [Ca2] [Ca3] [CJK] [CKS]

[CS]

[CSh] [CH]

H. W. ALT, L. A. CAFFARELLI, and A. FRIEDMAN, Variational problems with two phases

and their free boundaries, Trans. Amer. Math. Soc. 282 (1984), 431 – 461. MR 85h:49014 3, 8 D. E. APUSHKINSKAYA and N. N. URALTSEVA, On the behavior of the free boundary near the boundary of the domain (in Russian), Zap. Nauchn. Sem. S.-Peterburg. Otdel. Mat. Inst. Steklov. (POMI) 221 (1995), 5 – 19; English translation in J. Math. Sci. (New York) 87 (1997), 3267 – 3276. MR 96m:35340 2 I. ATHANASOPOULOS and L. A. CAFFARELLI, A theorem of real analysis and its application to free boundary problems, Comm. Pure Appl. Math. 38 (1985), 499 – 502. MR 86j:49062 23 I. ATHANASOPOULOS, L. A. CAFFARELLI, and S. SALSA, The free boundary in an inverse conductivity problem, J. Reine Angew. Math. 534 (2001), 1 – 31. MR 2002d:35219 3 H. BERESTYCKI, L. A. CAFFARELLI, and L. NIRENBERG, “Uniform estimates for regularization of free boundary problems” in Analysis and Partial Differential Equations, Lecture Notes in Pure and Appl. Math. 122, Dekker, New York, 1990, 567 – 619. MR 91b:35112 2 H. BERESTYCKI and B. LARROUTUROU, A semi-linear elliptic equation in a strip arising in a two-dimensional flame propagation model, J. Reine Angew. Math. 396 (1989), 14 – 40. MR 91a:35071 2 H. BERESTYCKI, B. LARROUTUROU, and L. NIRENBERG, “A nonlinear elliptic problem describing the propagation of a curved premixed flame” in Mathematical Modeling in Combustion and Related Topics (Lyon, 1987), NATO Adv. Sci. Inst. Ser. E Appl. Sci. 140, Nijhoff, Dordrecht, 1988, 11 – 28. MR 91a:35070 2 L. A. CAFFARELLI, The regularity of free boundaries in higher dimensions, Acta Math. 139 (1977), 155 – 184. MR 56:12601 2, 3 , Compactness methods in free boundary problems, Comm. Partial Differential Equations 5 (1980), 427 – 448. MR 81e:35121 3, 11, 12 , The obstacle problem revisited, J. Fourier Anal. Appl. 4 (1998) 383 – 402. MR 2000b:49004 2, 3, 7, 12, 23 L. A. CAFFARELLI, D. JERISON, and C. E. KENIG, Ann. of Math. (2) 155 (2002), 369 – 404. CMP 1 906 591 9, 32 L. A. CAFFARELLI, L. KARP, and H. SHAHGHOLIAN, Regularity of a free boundary with application to the Pompeiu problem, Ann. of Math. (2) 151 (2000), 269 – 292. MR 2001a:35188 2, 3, 4, 6, 10, 11, 12, 13, 23, 25, 29, 30 L. CAFFARELLI and J. SALAZAR, Solutions of fully nonlinear elliptic equations with patches of zero gradient: Existence, regularity and convexity of level curves, Trans. Amer. Math. Soc. 354 (2002), 3095 – 3115. CMP 1 897 393 3 L. A. CAFFARELLI and H. SHAHGHOLIAN, The structure of the singular set of a free boundary in potential theory, 2000, unpublished. 7, 8, 24 J. R. CANNON and C. D. HILL, On the movement of a chemical reaction interface, Indiana Univ. Math. J. 20 (1970/1971), 429 – 454. MR 43:5170 3

34

[ESS]

[F] [GT]

[Gu] [KS] [KN] [KSt]

[LU]

[U1]

[U2]

[U3]

SHAHGHOLIAN and URALTSEVA ¨ C. M. ELLIOTT, R. SCHATZLE, and B. E. E. STOTH, Viscosity solutions of a degenerate

parabolic-elliptic system arising in the mean-field theory of superconductivity, Arch. Ration. Mech. Anal. 145 (1998), 99 – 127. MR 2000j:35112 3 A. FRIEDMAN, Variational Principles and Free-Boundary Problems, Pure Appl. Math., Wiley, New York, 1982. MR 84e:35153 2 D. GILBARG and N. S. TRUDINGER, Elliptic Partial Differential Equations of Second Order, 2d ed., Grundlehren Math. Wiss. 224, Springer, Berlin, 1983. MR 86c:35035 A. GUREVICH, Boundary regularity for free boundary problems, Comm. Pure Appl. Math. 52 (1999), 363 – 403. MR 99m:35272 2 L. KARP and H. SHAHGHOLIAN, Regularity of a free boundary problem, J. Geom. Anal. 9 (1999), 653 – 669. MR 2001f:35444 3, 12 D. KINDERLEHRER and L. NIRENBERG, Regularity in free boundary problems, Ann. Scuola Norm. Sup. Pisa Cl. Sci. (4) 4 (1977), 373 – 391. MR 55:13066 5 D. KINDERLEHRER and G. STAMPACCHIA, An Introduction to Variational Inequalities and Their Applications, Pure Appl. Math. 88, Academic Press, New York, 1980. MR 81g:49013 3 O. A. LADYZHENSKAYA and N. N. URALTSEVA, Estimates on the boundary of the domain of first derivatives of functions satisfying an elliptic or a parabolic inequality (in Russian), Trudy Mat. Inst. Steklov. 179 (1988), 102 – 125; English translation in Proc. Steklov Inst. Math. 1989, no. 2, 109 – 135. MR 89k:35083 27 N. N. URALTSEVA, C 1 regularity of the boundary of a noncoincident set in a problem with an obstacle (in Russian), Algebra i Analiz 8, no. 2 (1996), 205 – 221; English translation in St. Petersburg Math. J. 8, no. 2 (1997), 341 – 353. MR 97m:35105 2, 3 , On the properties of a free boundary in a neighborhood of the points of contact with the known boundary (in Russian), Zap. Nauchn. Sem. S.-Peterburg. Otdel. Mat. Inst. Steklov. (POMI) 249 (1997), 303 – 312; English translation in J. Math. Sci. (New York) 101 (2000), 3570 – 3576. MR 2000d:35266 2 , On the contact of a free boundary with a given boundary (in Russian), Mat. Sb. 191, no. 2 (2000), 165 – 173. MR 2001i:35307 2

Shahgholian Department of Mathematics, Royal Institute of Technology, S-100 44 Stockholm, Sweden; [email protected] Uraltseva Department of Mathematics and Mechanics, St. Petersburg State University, 198904 St. Petersburg, Russia; [email protected]

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I ´ MARIE-FRANCE VIGNERAS

Abstract We give a link—through the affine Schur algebra—between the representations of the p-affine Schur algebra of GL(n) over R and the smooth R-representations of the p-adic group GL(n, Q p ) over any algebraically closed field R of characteristic not equal to p. Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2. The Schur algebra of a reductive p-adic group . . . . . . . 3. Morita equivalences . . . . . . . . . . . . . . . . . . . . 4. Decomposition of the parahoric restriction-induction functor 5. Proof of the main theorem . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35 42 45 47 62 73

1. Introduction There are relations between the representations of the following groups: the p-adic linear groups, the p-adic Galois groups, the p-affine Schur algebras, and the linear p-affine quantum groups. We wish to consider representations on any algebraically closed field R of characteristic not equal to p, called R-representations or modular representations. The case where the characteristic of R is p remains mysterious. The R-representations of the p-affine Schur algebras are the rational representations of the linear p-affine quantum groups over R. They appear as the nilpotent part of the Jordan decomposition of the R-representations of the p-adic linear groups. The semisimple part is given by the semisimple R-representations of the p-adic Galois groups. The complete local Langlands R-correspondence gives a parametrization of the irreducible R-representations for the linear p-adic groups (see [V5]), but here we consider the category of all R-representations, not only the irreducible ones. The DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 1, Received 5 April 2001. Revision received 6 November 2001. 2000 Mathematics Subject Classification. Primary 22E50. Author’s work supported by the Mathematical Sciences Research Institute and the Centre de la Recherche Scientifique. 35

´ MARIE-FRANCE VIGN ERAS

36

main new result is the main theorem in Section 1.2, which extends to R a theorem of J. Bernstein, A. Borel, and W. Casselman in the complex case. The main tool is the parahoric induction and restriction for a general reductive p-adic group and for the corresponding Iwahori Hecke algebra. 1.1 We have the following. F is a local nonarchimedean field of residual characteristic p, and q is the order of the residual field. F is an algebraic separable closure of F. R is an algebraically closed field of characteristic zero or ` 6= p. n is an integer greater than or equal to 1. Mod R G is the category of smooth R-representations of G := GL(n, F), equivalent to the category of right nondegenerate modules of the global Hecke R-algebra H R (G) of G isomorphic to its opposite algebra (see [V1, I.4.4]) Mod R G ' Mod H R (G). The following theorem is a formulation of the local Langlands R-correspondence. THEOREM

(a) (b)

(c)

(d)

The category Mod R G = 5τ B R,τ is a direct sum of blocks. The blocks B R,τ are parametrized by the semisimple R-representations τ of dimension n of the inertia subgroup I F of the Galois group of F/F, stable by the Frobenius of Gal(Fq /Fq ). The irreducible R-representations of the unipotent block B R,1 are parametrized by the pairs (s, N ), where s ∈ GL(n, R) is semisimple, N ∈ M(n, R) is nilpotent, and s N = N sq, modulo isomorphism. For any τ , the irreducible R-representations of the block B R,τ are parametrized by the irreducible R-representations of the unipotent block of a product of linear p-adic groups Y Gτ = GL(m τ,σ , Fn(σ ) ) σ

for all orbits σ of the Frobenius in the irreducible R-representations of I F occurring in τ , where • m τ,σ is the multiplicity of any irreducible constituent of σ in τ , • Fn(σ ) ⊂ F is the unramified extension of F of degree number n(σ ) of elements of σ .

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

37

The block B R,1 containing the trivial representation and the representations of B R,1 is called unipotent. This is consistent with the terminology of G. Lusztig in the complex case. Example We deduce from Theorem 1.1(c) the number m(s) of irreducible unipotent Rrepresentations with a given semisimple s ∈ GL(n, R). (a) If s = 1n is the identity, we have m(1n ) = 1 corresponding to N = 0 when q 6= 1 in R. But if q = 1 in R, that is, if the characteristic of R is ` 6= p and q = 1 mod `, then m(1n ) is the number p(n) of partitions of n. (b) If s is the diagonal (1, q, q 2 , q 3 ), the number m(s) depends on the multiplicative order e of q in R. When e = 1, 2, 3, 4, > 4 we have, respectively, m(s) = 5, 10, 9, 15, 8 (see [V2]). 1.2 Let I be an Iwahori subgroup of G, and let J R be the annihilator in H R (G) of the R-representation of G, R[I \G]. The category B 0R,1 of R-representations of G annihilated by J R is an abelian subcategory of the unipotent block B R,1 . MAIN THEOREM

There exists an integer N such that J RN annihilates the unipotent block. The category B 0R,1 is equivalent to the category of modules of the affine Schur algebra S R of G. One can give an explicit value for N . One can replace the ideal J R by its component J R,1 in the unipotent part of the global Hecke algebra H R (G). The ideal J R,1 is N = {0}. We define the affine Schur algebra S of G in Section 1.3. nilpotent, J R,1 R The Iwahori-Hecke algebra H R of G is the ring of endomorphisms of R[I \G], H R := End RG R[I \G].

It would be a natural candidate for the affine Schur algebra, modulo Morita equivalence. But it does not work because there are not enough simple H R -modules. The irreducible quotients of R[I \G] are in bijection with the simple H R -modules. But a subquotient is not always a quotient. An example is GL(2, Q5 ) when the characteristic of R is 3. The irreducible subquotients of R[I \G] are the irreducible unipotent R-representations of G (see [V3, II.11.2]). They are also the irreducible Rrepresentations of G annihilated by J R , and they are in bijection with the simple S R -modules.

´ MARIE-FRANCE VIGN ERAS

38

1.3 Definition The Schur algebra S R of G is the ring of endomorphisms of the right H R -module V RI , S R := EndH R V RI , (1.1) where V R is the R-representation of G, V R :=

M

R[P\G]

(1.2)

P

for all parahoric subgroups P of G containing I (called standard). We have a natural inclusion R[P\G] ⊂ R[I \G] in Mod R G; hence the annihilator of V R is J R , and the irreducible subquotients of V R are the irreducible unipotent representations of G. We have
HomH R R[I \G/I ], R[P\G/I ] ' R[P\G/I ]. (1.3) 1.4 PROPOSITION

The S R -module V RI is cyclic generated by the projector e : V RI → R[I \G/I ],
V RI ' S R e = HomH R R[I \G/I ], V RI , (1.4) and the (S R − H R )-module V RI satisfies the double centralizer property (see (1.1)) eS R e = H R = EndS R V I .

(1.5)

The simple H R -modules are in bijection with the simple S R -modules W such that eW 6= 0. There is another property that is not so evident: the Schur algebra is isomorphic to the ring of RG-endomorphisms of V R (Theorem 2.3 proved in Section 4.2.14), End RG V R ' EndH R V RI .

(1.6)

This is valid when GL(n, F) is replaced by any reductive connected group over F (the group of F-points) and R is replaced by any commutative ring. The Iwahori-Hecke R-algebra and the Schur R-algebra are naturally defined over Z. They are free Z-modules with natural basis and S R = SZ ⊗Z R, and the same is well known for H R .

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

39

1.5 The main theorem is interesting because it creates a bridge between the p-adic groups and the affine quantum groups. Indeed, [Gre] introduces an affine quantum group UC (n, q) for gl(n) over C (a Hopf algebra) and a natural UC (n, q)-module W of infinite countable dimension such that the natural action of UC (n, q) on the tensor space W ⊗n has the following properties. • The image SC (n, q) of UC (n, q) in End R W ⊗n is isomorphic to the Schur algebra SC . • The centralizer of the action of UC (n, q) on W ⊗n is isomorphic to the IwahoriHecke algebra HC . • The tensor space W ⊗n is isomorphic to V I as an (SC − HC )-module. One hopes that there are natural integral structures over A = Z[v, v −1 ] such that the image of U A (n, q) in End A W ⊗n is isomorphic to S A . When the characteristic of R is greater than zero and q 6= 1 in R, if ζ is a root of 1 in C∗ of the same order as q in R ∗ , one hopes to better understand problems as decomposition numbers or as the cohomology of modular representations of G with the help of the complex affine quantum group UC (n, ζ ). There are similar theorems and applications for finite linear groups GL(n, Fq ) by R. Dipper and G. James [DJ1], [DJ2], M. Takeuchi [T], and E. Cline, B. Parshall, and L. Scott [CPS]. The proof of the main theorem does not rely on the theorem in the finite case. 1.6 The proof of the main theorem is a generalization of the proof of Takeuchi [T] for GL(n, Fq ) and is given in Section 5. It is more conceptual and also gives a new proof in the finite case. The idea is the following. As in Takeuchi, one considers the group U ∗ (Fq ) of (ai j ) ∈ GL(n, Fq ) with ai j = 0 if i > j or j = i + 1 and aii = 1 for all i, j such that this has a sense. Let U ∗ be the inverse image of U ∗ (Fq ) by the reduction modulo p, GL(n, O F ) → GL(n, Fq ), where O F is the ring of integers of F. Set G := GL(n, F). One considers the right H R -module R[U ∗ \G/I ]. Its ring of endomorphisms is Morita equivalent to the Schur algebra (Proposition 5.5) EndH R R[U ∗ \G/I ] 'Morita S R , and R[U ∗ \G/I ] are the I -invariants of the R-representation of G, R[U ∗ \G] = e∗ H R (G),

´ MARIE-FRANCE VIGN ERAS

40

which is cyclic and projective, generated by the idempotent e∗ defined by the pro- pgroup U ∗ . One shows that the algebra homomorphism of I -invariants End RG e∗ H R (G) → EndH R e∗ H R (G) I is surjective in Lemma 5.6, from which one deduces in Proposition 5.8 that
e∗ H R (G)/J R e∗ 'Morita S Ro . It remains to prove that the functor
Hom e∗ H R (G)/J R , − : Mod H R (G)/J R → Mod S R defines an equivalence of categories (Proposition 5.10). At this point, the fact that we are working with a linear group is crucial. The theory of the Whittaker model for GL(n, Fq ) shows that any irreducible R-representation of G annihilated by J R has a nonzero vector invariant by U ∗ . This implies the equivalence of categories and also implies that the unipotent part e∗ H R (G)1 of e∗ H R (G) is a progenerator of the unipotent block. The composition of the parabolic induction functor in the finite group GL(n, Fq ) with the inflation to GL(n, O F ) and the compact induction to G = GL(n, F) is a functor called parahoric induction. The parahoric induction commutes with the functor unipotent part (Lemma 5.14). From this, one deduces in Theorem 5.13 that M G e∗ H R (G)1 ' iM 0 J R,J,1 J

is a direct sum of representations parahorically induced from the standard Levi subgroups M J of GL(n, Fq ), where 0 R,J,1 is the unipotent part of the Gelfand-Graev representation of M J (with some multiplicity). When M J is the diagonal torus T (Fq ), R[I \G] = i TG(Fq ) 1 R . The transitivity of the parahoric induction shows that e∗ H R (G)1 , hence any unipotent representation of G, has a finite filtration with quotients isomorphic to subquotients of R[G/I ]. The length N of the filtration is bounded by the maximum of the analogous filtrations of 0 R,J,1 with I replaced by a minimal parabolic subgroup of M J for all standard Levi subgroups M J of GL(n, Fq ). 1.7 One can compare J R with the intersection J R in H R (G) of the annihilators of the irreducible unipotent R-representations of G. The unipotent part J R,1 of J R is the

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

41

Jacobson radical of the unipotent part of H R (G). The representation R[I \G] is isomorphic to R[I \G] ' i TG R[T (O F )\T ], where T = T (F) is the diagonal group with the universal action on R[T (O F )\T ] and i TG is the nonnormalized parabolic induction from T to G (see [D]). The theory of R-types for GL(n, F) shows that the lengths of the induced representations i TG χ for all R-characters χ of T trivial on T (O F ) are bounded by the length of i TG 1 R (see [V3]). This property and the main theorem imply the following. PROPOSITION

Let M be the length of i TG(F) 1 R . Then J RM ⊂ J R ⊂ J R . A finite power of the Jacobson radical of the unipotent part of H R (G) is zero. 1.8 Although the main theorem concerns only p-adic general linear groups and algebraically closed fields of characteristic different from p, we develop a theory for a general reductive p-adic group G and for any commutative ring R ( p can be zero in R). The main theorem is proved for a general reductive p-adic group G in the linear case when the finite reductive quotients of the parahoric subgroups of G are products of linear groups, in particular, when G = GL(n, D) where D/F is a division algebra over F. Section 2 introduces the Schur algebra of a reductive p-adic group, and Section 3 recalls some properties of the functor Hom A (Q, −) when Q is a quasi-projective module over an algebra A. In Section 4 one studies R-representations induced from the finite reductive quotient of a parahoric subgroup and their endomorphism rings. The Mackey decomposition of the parahoric restriction-induction plays a fundamental role. It is given in Theorem 4.1.4 for the group and in Theorem 4.2.1 for the IwahoriHecke algebra. One deduces a basis and relations for the Schur algebra of a reductive p-adic group in Section 4.2.15 with some restrictions on R. A systematic use of the parahoric induction allows us to reduce problems concerning representations of a reductive p-adic group G to problems concerning representations of the finite reductive quotients of the parahoric subgroups. It is particularly useful to study the functor of I -invariants linking R-representations of G and right modules for the Iwahori-Hecke R-algebra, as in Section 4.3.

42

´ MARIE-FRANCE VIGN ERAS

Remark The properties of the generalized BN-pair and of the parahoric subgroups in a general reductive group G that we need are all listed in Sections 2.1 and 4.1.1. The reader should be aware that they are extracted from statements (sometimes without ´ proofs and with different parametrizations) in the two Institut des Hautes Etudes Scientifiques (IHES) articles by F. Bruhat and J. Tits [BT1], [BT2], in the Corvallis summary of Tits [Ti], and in different articles of L. Morris [M1], [M2]. It is safer to say that the results presented here are proved for reductive groups for which the list is valid. 1.9 The main theorem can be generalized to other blocks using the theory of BushnellKutzko types. One can also show that it is compatible with the reduction modulo `. In more concrete terms, the decomposition numbers of GL(n, F) are decomposition numbers for the affine Schur algebras, and the converse is also true. This will be explained in part II. 2. The Schur algebra of a reductive p-adic group Let R be a commutative ring. The q-affine deformations of the group algebra RSn of the symmetric group Sn on n elements, of the tensor space V ⊗n where V = R n , and of the Schur algebra End R Sn V ⊗n admit obvious analogues for a reductive connected p-adic group G. They are the Iwahori-Hecke algebra of G, the tensor space of G, and the Schur algebra of G which we define in this section. 2.1 We introduce some notation and definitions (see the caution remark in Section 1.8) following Bruhat and Tits [BT1], [BT2] and Morris [M1], [M2]. We denote the following. F is a local nonarchimedean field with residual field Fq with q = p f . G is the group of F points of a connected reductive group G over F. T is the group of F points of a maximal F-split torus T of G. Z is the group of F points of the centralizer Z of T in G. N is the group of F points of the normalizer N of T in G. W := N/Z ' N /Z is the Weyl group of (G, T ). 5 is a basis of the affine simple roots defined by (G, T ). J ⊂ 5 is a proper subset different from 5. PJ is a standard parahoric subgroup (see [BT2, 5.2.4, 5.2.6], [M1, 3.14]; an Iwahori subgroup P∅ = I when J is the empty set. A parahoric subgroup is conjugate to a standard parahoric subgroup, but we consider only a standard parahoric subgroup,

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

43

and we omit “standard.” The parahoric subgroup PJ has a pro- p-radical U J of quotient M J := PJ /U J , the group of Fq points of a connected reductive group M J over Fq , a torus when J = ∅. The pro- p-radical U J and the Levi component M J are as in [M1, 3.13]. Waff (G) = N /(I ∩ N ) is the generalized affine Weyl group of G; we have I ∩ N = I ∩ Z . The group Waff (G) is the semidirect product of W and of Z /(I ∩ Z ). The map n → I n I induces a bijection from Waff (G) and the double (I, I )-cosets of G (see [M1, 3.22]). One denotes also by I w I := I n I the coset associated to the image w ∈ Waff (G) of n ∈ N . The generalized affine Weyl group is the group denoted by W in [M1, 3.2]. The group N does not have the same definition, but the two definitions coincide by [Ti, 2.1]. si , i ∈ 5 is the element of order 2 in Waff (G) such that the Pi := I ∪ I si I . W J is the subgroup of Waff (G) generated by the affine reflections s j , j ∈ J . The group W J is as in [M1, 3.1(c)] and is the group W J0 of [M1, 3.9]. The subgroup of W given by the vector parts of W J is isomorphic to W J and to the Weyl group of M J . We identify them. The group W∅ is trivial. We have PJ = I W J I . For J, K two proper subsets of 5, the map n → PJ n PK induces a bijection (see [M1, 3.22]) W J \Waff (G)/W K ' PJ \G/PK . Mod R G is the category of smooth R-representations of G; Irr R G is the category of irreducible ones. All representations are smooth. 2.2 The q-affine deformation of R Sn , or the q-deformation of R[Waff (G)], is the IwahoriHecke R-algebra of G, H R (G, I ) := End RG R[I \G].

The R-representation of G, M V R (G) := V R,J ,

V R,J := R[PJ \G]

J

is a natural candidate for the tensor R-space of G in the category Mod R G. The invariant of V R (G) by the Iwahori subgroup I , M V R (G) I ' R R[PJ \G/I ], J

is the natural candidate for the tensor space of G in the category Mod H R (G, I ) of right H R (G, I )-modules. The Schur R-algebra S R (G) of G defined by the tensor space V R (G) ∈ Mod R G is the endomorphism algebra S R (G) := End RG V R (G).

´ MARIE-FRANCE VIGN ERAS

44

The Schur algebra defined by the tensor space V R (G) I ∈ Mod H R (G, I ) is the endomorphism algebra S R (G, I ) := EndH R (G,I ) V R (G) I . The tensor space V R (G) I is a deformation of the R[Waff (G)]-module I I idempotents of the J R[W J \Waff (G)]. The projectors e J : V R (G) → V R,J are P Schur algebra S R (G, I ) satisfying e J e K = 0 for J 6= K and J e J = 1 (the unity of S R (G, I )). We simply write e = e∅ . It is clear that
e J S R (G, I )e ' HomH R (G,I ) R[I \G/I ], R[PJ \G/I ] . L

We deduce that the S R (G, I )-module V R (G) I is cyclic, V R (G) I ' S R (G, I )e, and that the (S R (G, I ) − H R (G, I ))-module V R (G) I satisfies the double centralizer property H R (G, I ) = EndS R (G,I ) V R (G) I . As a particular case, we get the first part of Proposition 1.4. We show in Section 4.2.14 that the Schur algebras S R (G) and S R (G, I ) are isomorphic. 2.3 THEOREM

For any commutative ring R and any reductive connected p-adic group G, the functor of I -invariants induces an algebra isomorphism End RG V R (G) ' EndH R (G,I ) V R (G) I . We exhibit a natural basis of End RG V R (G), as an R-module, corresponding by the functor of I -invariants to the natural basis of EndH R (G,I ) V R (G) I . 2.4 Remarks (1) It is clear that the Iwahori-Hecke algebra, the tensor space, and the Schur algebra are defined over Z. For any of these objects X R , one has X R = X Z ⊗Z R. (2)

We may get rid of the redundancies in the tensor space V R (G) in order to define the Schur algebra. This results from Section 3.6.

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

45

3. Morita equivalences Let A be an algebra. We denote by Mod A the abelian category of nondegenerate right A-modules M = M A (we do not suppose that A contains a unit) and by A Mod the nondegenerate left A-modules, that is, the nondegenerate right modules for the opposite algebra Ao . Let Q ∈ A Mod. We recall in this section some known properties of the functor Hom(Q, −) from A Mod to the category of right End A Q-modules. 3.1 We recall some definitions. Let B be another algebra. One says that A, B are Morita right equivalent if the categories Mod A and Mod B are equivalent, Morita left equivalent if the categories A Mod and B Mod are equivalent, and Morita equivalent if A, B are both left and right Morita equivalent. Q ∈ Mod A is called a progenerator when it is projective and finitely generated and when any right A-module is the quotient of a direct sum of representations isomorphic to Q. Q ∈ Mod A is quasi projective when, for each pair of homomorphisms f, g : Q → V in Mod A with g surjective, there exists φ ∈ End A Q with f = g ◦ φ. The same definitions can be given for Q ∈ A Mod. 3.2 Example For (G, I, R) as in Section 2, when p is invertible in R, the cyclic R-representation of G, Q = R[I \G] is quasi projective (see [V3, Proposition I.3]). One cannot replace the Iwahori group I by another parahoric subgroup. Moreover, Q is projective when the pro-order of I is invertible in R. 3.3 THEOREM

Let Q ∈

A

Mod be quasi projective and finitely generated. The functor Hom A (Q, −) :

A

Mod → Mod End A Q

induces the following: (1) a bijection between the simple quotients of Q and the simple right End A Qmodules, N (2) an equivalence of categories with inverse the functor − A Q when Q is projective and Hom A (Q, V ) 6= 0 for any simple V ∈ A Mod. Then Q is a progenerator of A Mod.

´ MARIE-FRANCE VIGN ERAS

46

See [V3, Appendix, Theorem 4(2)]. Property (2) results from [P, §3.7, Theorem 7.6] using the following. 3.4 CLAIM

When Q ∈ A Mod is projective and finitely generated and Hom A (Q, V ) = 6 0 for P any simple V ∈ A Mod, then any V ∈ A Mod is the sum f f (Q) for all f ∈ Hom R (Q, V ). Proof (a) Let X ∈ A Mod be nonzero. Let W be a nonzero finitely generated subrepresentation of X , and let W1 be an irreducible quotient of W . By hypothesis, Hom A (Q, W1 ) 6= 0. By hypothesis, Q is projective; hence Hom A (Q, W ) 6= 0. Hence Hom A (Q, X ) 6= 0. P (b) Let V ∈ A Mod be nonzero. Let V 0 := f f (Q) for all f ∈ Hom R (Q, V ). We have Hom A (Q, V ) = Hom A (Q, V 0 ). The functor Hom A (Q, −) is exact; hence Hom A (Q, X ) = 0 for X = V /V 0 . By (a), V = V 0 . 3.5 Example Suppose that Q = Ae is generated by an idempotent e = e2 of A. The value at e gives an isomorphism of abelian groups Hom A (Ae, V ) ' eV for all V ∈ A Mod. When V = Ae, the value at e is an isomorphism of algebra (End A Ae)o ' e Ae. The functor V → eV : A Mod → e Ae Mod is an equivalence of categories if eV 6= 0 for any nonzero simple V ∈ A Mod. 3.6 Let M1 , . . . , Mn ∈ A Mod; there can be redundancy. We suppose that M1 , . . . , Mk are not isomorphic and that for each 1 ≤ i ≤ n there is some 1 ≤ j ≤ k such that Mi is isomorphic to M j . We set B := End A M,

M := M1 ⊕ · · · ⊕ Mn ,

C := End A N ,

N := M1 ⊕ · · · ⊕ Mk .

The natural projection e : M → N is an idempotent in B, and eBe = C. We show that the algebras B, C are Morita equivalent.

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

47

CLAIM

The functor V → eV : B Mod → C Mod and the functor V → V e : Mod B → Mod C are equivalences of categories. Proof The proofs are symmetrical. Let us consider the functor V → eV . By Example 3.5, it is enough to prove that eV 6= 0 for any nonzero V ∈ B Mod. The projections ei : M → Mi for 1 ≤ i ≤ n are orthogonal idempotents in B with sum the unit 1 of B, and e1 + · · · + ek = e. There exists some 1 ≤ i ≤ n with ei V 6= 0. We prove that e j V 6= 0 for all j such that M j ' Mi . This implies eV 6= 0. One reduces easily to the case where all Mi are isomorphic, that is, where k = 1. Then B ' M(n, C) and the idempotent ei identifies to a diagonal matrix with the single nonzero entry equal to 1 at (i, i). The idempotents are permuted by the symmetric group naturally embedded in the group B ∗ of invertible elements of B; hence for any 1 ≤ i ≤ n there exists a unit b ∈ B ∗ such that e1 = bei b−1 . If V is a nonzero B-module, then ei V 6= 0 for some i and e1 V = bei b−1 V 6= 0. We need the following property deduced from Claim 3.4 and from [P, §3.7, Theorem 7.9]. 3.7 CLAIM

Let Q ∈ Mod A be finitely generated with annihilator J in A. When Q is projective in Mod A/J and Hom A (Q, V ) 6= 0 for any simple V ∈ Mod A/J , the functor Hom A (Q, −) : Mod A/J ' Mod(End A Q)o is an equivalence of categories, with inverse the functor − a progenerator of Mod A/J .

N

A/J

Q. Moreover, Q is

4. Decomposition of the parahoric restriction-induction functor We suppose that R is any commutative ring. We do not suppose that p is invertible in R in view of future applications to the case where the characteristic of R is p. The parahoric restriction and induction functors for the groups in Section 4.1 are defined in [Vig6], where they play a fundamental role in the classification of the irreducible R-representations of G of level zero when R is algebraically closed. They appear naturally with the Schur algebra. The decomposition of the restriction-induction functor is very useful for studying the functor I -invariants or for getting a natural basis for the affine Schur algebra. The analogue for the Hecke algebras is given in Section 4.2, and the comparison via the invariant functors is done in Section 4.3. We start in Section

´ MARIE-FRANCE VIGN ERAS

48

4.1.1 with a list of properties extracted from Bruhat and Tits [BT1], [BT2] and from Morris [M1] (see Remark 1.8). 4.1. Groups 4.1.1. Distinguished elements We need even more notation than in Section 2. The situation is complicated by the fact that the generalized affine Weil group Waff (G) is not a Coxeter group (and we insist on working on a general reductive p-adic group). We denote the following. G 0 is the normal subgroup of G generated by the parahoric subgroups of G. N 0 := G 0 ∩ N , and the double (I, I )-cosets of G contained in G 0 are I n 0 I for 0 n ∈ N 0. Waff := N 0 /(I ∩ N 0 ) is the affine Weyl group of the affine B N -pair (G 0 , I, N 0 ); it is a Coxeter group for (si )i∈5 .  ∈ Waff (G) is the subgroup such that the double (I, I )-cosets of G contained in the normalizer of I in G are I w I, w ∈ . The group  is abelian, is isomorphic to N /N 0 ' G/G 0 , and normalizes I and (si )i∈5 . The generalized affine Weyl group Waff (G) is a semidirect product of the normal subgroup Waff and of . ` is the length on Waff extended to a length on Waff (G) such that the set of elements of Waff (G) of length zero is equal to . J or K is a proper subset of 5. A set of representatives n w , w ∈ Waff (G) in N of Waff (G) can be chosen such that n w1 w2 = n w1 n w2 when `(w1 w2 ) = `(w1 ) + `(w2 ) and n w ∈ N 0 if w ∈ Waff (see [M1, 5.2]). We fix such a set, and we often identify n w with w. Any double (W K , W J ) coset in W K \Waff (G)/W J contains a unique element of minimal length, called distinguished by (K , J ). The set of these elements is denoted by D K ,J (G). It is identified with its representatives in N . Clearly, D J,K (G) (in Waff (G)) is the set D K ,J (G)−1 of inverse elements of D K ,J (G). The same is true for the representatives of D J,K (G) in N . When L is another proper subset of 5 which contains K ∪ J , it is clear that D K ,J (G) ∩ W L is the set of (K , J )-distinguished elements of W L . It is denoted by D KL ,J . Let d ∈ D K ,J (G). We denote by Int(d) the conjugation by d when it has a sense. The important properties of d are the following. (a) For all w ∈ W J and d 0 ∈ D J,∅ , we have (see [M1, 3.9]) `(w J d 0 ) = `(w J ) + `(d 0 ). For w K ∈ W K , w J ∈ W J , the relation `(w K dw J ) = `(w K ) + `(d) + `(w J ) is not always true (there is no addition of the lengths in W K d W J ), but one has (there is no reference in Morris, but the proof is as in [C, 2.7.5] using [M1, 3.16, 3.17]) K W K d W J = D∅,K ∩ d J d WJ

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

49

K with unique decomposition w = adw J , a ∈ D∅,K ∩ d J , w J ∈ W J , w ∈ W K d W J , and K there is addition of the lengths in D∅,K ∩ d J dW J for `(adw J ) = `(a) + `(d) + `(w J ); that is, ad ∈ D∅,J . (b) By [C, 2.7.4] and [M1, 3.17],

W K ∩ d W J d −1 = W K ∩ d J . The map v → dvd −1 : Wd −1 K ∩ J → W K ∩ d J is an isomorphism. (c) By [C, 2.8.7] and [M1, 3.19, 3.20, 3.21], the group PK ∩Int(d)PJ modulo U K is a standard parahoric subgroup PKK∩ d J of M K with unipotent radical U KK∩ d J equal to PK ∩ Int(d)U J modulo U K . The finite reductive quotient of PKK∩ d J is denoted M KK∩ d J . (d) Suppose K ⊂ J . In the Bruhat-Tits semisimple building, the facet F J associated to J is contained in the closure of the facet F K associated to K . This implies that J → PJ respects the inclusions and that J → U J reverses the inclusions (see [BT2, 5.2.4], [BT1, 6.4.9, 7.1.1], [SS, I.2.11]): U J ⊂ U K ⊂ PK ⊂ PJ . The groups PK , U K are the inverse images of PKJ , U KJ in PJ by the reduction modulo U J , and we have a canonical isomorphism M K = PK /U K ' (PK /U J )/(U K /U J ) = PKJ /U KJ = M KJ . In particular, M KK∩ d J ' M K ∩ d J is identified. (e) The conjugation by d gives an isomorphism Int(d) : Int(d −1 )PJ → PJ respecting the pro- p-radicals and, by quotient, an isomorphism between the reductive quotients Int(d) : Md −1 J → M J . (f) The group B J := P∅J is a minimal parabolic subgroup of M J with unipotent radical U∅J . 4.1.2. Parahoric induction and restriction Let R be any commutative ring, and let Mod R G be the category of representations of G. We consider the following functors. (a) Representations of the finite reductive group M J identify with representations of the parahoric group PJ trivial on U J by inflation PJ infl M : Mod R M J → Mod R PJ . J

(b)

The right adjoint of the inflation is the U J -invariant functor PJ inv M : Mod R PJ → Mod R M J . J

´ MARIE-FRANCE VIGN ERAS

50

(c)

The compact induction is from representations of the parahoric group PJ to representations of the p-adic reductive group G, indGJ : Mod R PJ → Mod R G.

(d)

The right adjoint of the compact induction is the restriction resGJ : Mod R G → Mod R PJ .

(e)

The parahoric induction i JG and its right adjoint the parahoric restriction r JG are PJ i JG := indGPJ ◦ infl M : Mod R M J → Mod R PJ → Mod R G, J PJ r JG := inv M ◦ resGPJ : Mod R G → Mod R PJ → Mod R M. J

(f)

Suppose K ⊂ J . Then M K is the Levi subgroup of the parabolic subgroup PKJ of M J . We have the functors of parabolic induction and its right adjoint the parabolic restriction between the finite reductive groups PJ

i KJ := ind MJJ ◦ infl MKK : Mod R M K → Mod R PKJ → Mod R M J , PK

r KJ :=

PJ inv MKK

◦ res MJJ : Mod R M J → Mod R PKJ → Mod R M K . PK

It is well known that the parabolic induction and restriction functors in the finite or p-adic case are transitive. The parahoric functors relate p-adic groups and finite groups. We consider the composition of a parahoric functor with a parabolic functor between finite reductive subgroups. 4.1.3. Transitivity CLAIM

When K ⊂ J , then i KG = i JG ◦ i KJ ,

r KG = r KJ ◦ r JG .

Proof By adjunction, it is enough to prove one of the two equalities. They are true because we have an exact sequence 1 → U J → U K → U KJ → 1. 4.1.4 THEOREM (see [V6]) For any commutative ring R, the parahoric restriction-induction functor

TKG,J = r KG ◦ i JG : Mod R M J → Mod R G → Mod R M K

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

51

is a direct sum TKG,J =

M d∈D K ,J (G)

FKd ,J

of functors FKd ,J isomorphic to i KK ∩ d J ◦ Int(d) ◦ r JJ∩ d −1 K : Mod R M J → Mod R M J ∩ d −1 K → Mod R M K ∩ d J → Mod R M K . The decomposition is given by the restriction to the U K -invariant functions with support PJ d −1 PK . The isomorphism for FKd ,J is obtained via the map f → φ(m) = f (d −1 m) modulo U JJ∩ d −1 K for m ∈ M K . The sum is infinite, but only finite reductive groups appear in the definition of the functors FKd ,J . The following are the three basic cases: (a) the parabolic restriction r KJ ' FK1 ,J when K ⊂ J, 1 when K ⊂ J, (b) the parabolic induction i KJ ' FJ,K d (c) the conjugation Int(d) ' Fd J,J when d J ⊂ 5. 4.1.5. Basic example For any R-representation V of the Iwahori subgroup I = P∅ trivial on its pro- pradical I p = U∅ , we have an isomorphism of I /I p -modules M Ip (indG Int w V. I V) ' G w∈Waff

As r KG is the right adjoint of i KG , we have Hom RG (R[PK \G], R[PJ \G]) ' R Hom R PK (1, TKG,J 1), and Theorem 4.1.4 gives the following basis of this R-module. 4.1.6 BASIS

We have Hom RG (R[PJ \G], R[PK \G]) ' R

M d∈D K ,J (G)

Rψ Kd ,J ,

where ψ Kd ,J sends the characteristic function of PJ to the characteristic function of PK d PJ .

´ MARIE-FRANCE VIGN ERAS

52

4.2. Hecke algebras Let R be any commutative ring. The affine Iwahori-Hecke algebra H R (G, I ) is denoted by H R . The double (I, I ) coset of w ∈ Waff (G) considered in H R is denoted by Tw . The product is the convolution product for the Haar measure normalized by I . This “non sense,” valid when R = Q is the field of rational numbers, allows us to compute the product when R = Z and by scalar extension for any R. The Hecke algebra H R (PJ , I ) is denoted by H R,J . It is a subalgebra of H R , M H R,J = RTw , w∈W J

and is naturally isomorphic to the Hecke algebra H R (M J , B J ). For K ⊂ J , we have H R,K ⊂ H R,J . Notation is as in Section 3 for modules. We consider the following functors. (a) The induction iH J : Mod H R,J → Mod H R ,

V → V ⊗H R,J H R .

The right adjoint∗ of the induction is the restriction

(b)

r JH : Mod H R → Mod H R,J . 4.2.1 THEOREM

For any commutative ring R, the parahoric restriction-induction functor TKH,J := r KH ◦ i H J : Mod H R,J → Mod H R → Mod H R,K is a direct sum

TKH,J =

M

F Kd ,J

d∈D K ,J

of functors F Kd ,J isomorphic to H ,J i KH∩,K d J ◦ Int(d) ◦ r J ∩ d −1 K : Mod H R,J → Mod H R,J ∩ d −1 K

→ Mod H R,d J ∩ K → Mod H R,K . The three basic functors are the following: (a) the restriction r KH ,J : Mod H R,J → Mod H R,K when K ⊂ J, (b) the induction i KH ,J : Mod H R,K → Mod H R,J when K ⊂ J, ∗ The

right adjoint of the restriction is HomH M (H M , −).

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

53

the conjugation Int(d) : Tw 7→ Tdwd −1 , Mod H R,J → Mod H R,d J when d J ⊂ 5. We need some preparation for the proof of Theorem 4.2.1, which is given in Section 4.2.12. (c)

4.2.2. Relations in Iwahori-Hecke algebras Let G be as in Sections 2.1 and 4.1. Change of notation: we identify the simple affine roots in 5 with the associated reflections. The Iwahori-Hecke algebras are described when G is a Chevalley group by N. Iwahori and H. Matsumoto [IM] and when G is semisimple by A. Borel [B]. The Iwahori-Hecke algebra H R (G 0 , I ) of G 0 is the algebra associated to the Coxeter system (Waff , 5) and to the constants qs := [I s I /I ] = q ds for all s ∈ 5 (see [Bo, Chapter 4, Exercises 23, 24]), where ds is the integer attached to s as in [Ti, 1.8.1, 2.4, 3.5.4]. When s, s 0 are conjugate in Waff (G), the constants are equal; that is, qs = qs 0 . When the torus I /I p = M∅ is split, that is, when G is residually split, all the integers ds are equal to 1. The order of a maximal unipotent subgroup of Ms is qs . For w ∈ Waff (G) we set qw := [I w I : I ] = q d . If w = σ s1 · · · sr , where σ ∈ , and s1 · · · sr is a reduced word in Waff , then d = Pr 0 t=1 dst . When w, w ∈ Waff (G) are conjugate, then qw = qw0 . In particular, qw = qw−1 . The relations between the Tw for w ∈ Waff are Tww0 = Tw Tw0

if `(ww0 ) = `(w) + `(w0 ) for all w, w0 ∈ Waff ,

Ts2 = (qs − 1)Ts + qs T1

for all s ∈ 5.

(4.2.3) (4.2.4)

The Iwahori-Hecke algebra H R = H R (G, I ) of G is a generalized affine Hecke L algebra. As  normalizes I , the R-submodule σ ∈ RTσ is a subalgebra isomorphic to the group algebra R[], and it is known that for all w ∈ Waff (G) and σ ∈ , Twσ = Tw Tσ ,

Tσ w = Tσ Tw .

(4.2.5)

Hence we have H R ' R[] · H R (G 0 , I ).

The definition of the length on Waff (G) and the relation (4.2.5) show that the relation (4.2.3) is valid for all w, w0 ∈ Waff (G). We have Tww0 = Tw Tw0 if there is addition of the lengths in ww0 .

´ MARIE-FRANCE VIGN ERAS

54

When p is invertible in R, Ts is invertible in H R by (4.2.4), Ts−1 = (1 − qs−1 )T1 + qs−1 Ts , and by (4.2.3) and (4.2.5), Tw is invertible in H R for all w ∈ Waff (G). 4.2.6. Characters The map w → qw extends to a character of H R called the index or trivial character. The map w → (−1)`(w) extends to a character of H R called the sign character. The same is true for the finite Hecke algebra H R,J . Set X X xJ = Tw , yJ = (−1)−`(wo J w) qwo J w Tw , w∈W J

w∈W J

where wo J is the longest element of W J (see [C, 2.211]). When J = ∅, then x∅ = y∅ = T1 . Note that, by additivity of the lengths, qwo J = qwo J w qw−1 = qw qwo J w and that, by symmetry, qwo J = qw qwwo J . If p is invertible in R, we have X y J = (−1)−`(wo J ) qwo J (−1)−`(w) qw−1 Tw . w∈W J

4.2.7 LEMMA

Modulo multiplication by an element of R, x J and y J are the unique elements in H R,J which satisfy the relations x J Ts = Ts x J = qs x J ,

y J Ts = Ts y J = −y J

for all s ∈ J . Proof Let aw ∈ R for w ∈ W J . Let s ∈ J . Then X  X aw Tw Ts = w∈W J

(aw Tw + aws Tws )Ts .

w∈W J , `(w)<`(ws)

We have, for w ∈ W J , `(w) < `(ws), (aw Tw + aws Tws )Ts = aw Tws + aws Tw qs T1 + (qs − 1)Ts




= qs aws Tw + (qs aws + aw − aws )Tws . The equations qs aw = qs aws ,

qs aws = qs aws + (aw − aws )

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

55

are equivalent to aw = aws . They are satisfied for all s ∈ J if and only if aw is constant for w ∈ W J . The equations −aw = qs aws ,

−aws = qs aws + (aw − aws )

are equivalent to aw = −qs aws . They are satisfied for all s ∈ J if and only if aw = −1 (−1)`(w wo J ) qw−1 wo J awo J for all w ∈ W J . We can replace w−1 wo J with its inverse wo J w in the term before awo J . The proof is the same for the other side. 4.2.8 LEMMA

(a)

When K ⊂ J, we have x K x J = x J x K = PK (q)x J ,

wher e PK (q) :=

X

qw .

w∈W K

(b) (c)

For d ∈ D K ,J (G), we have Pd −1 K ∩ J (q) = PK ∩ d J (q). For d ∈ D K ,J (G), we have X x K Td x J = PK ∩ d J (q) Tw . w∈W K d W J

Proof Part (a) results from Lemma 4.2.7, part (b) results from Section 4.1.1(b), and qw = qw0 when w, w0 ∈ Waff (G) are conjugate. K We prove (c). We have W K dW J = D∅,K ∩ d J dW J with addition of the lengths on the right side from Section 4.1.1(a). Hence X X Tw = Tw Td x J . w∈W K d W J

K w∈D∅,K ∩dJ

K We have W K = D∅,K ∩ d J W K ∩ d J with addition of the lengths from Section 4.1.1(a). Hence X xK = Tw x K ∩ d J . K w∈D∅,K ∩dJ

We have W K ∩ d J d = dWd −1 K ∩ J with addition of the lengths; hence X X x K Td = Tw x K ∩ d J Td = Tw Td xd −1 K ∩ J . K w∈D∅,K ∩dJ

K w∈D∅,K ∩dJ

By right multiplication by x J , using (a) and (b), we get the formula.

´ MARIE-FRANCE VIGN ERAS

56

4.2.9. Iwahori-Matsumoto automorphism CLAIM

Suppose that p is invertible in R. (a) There is an automorphism j of order 2 of the Iwahori-Hecke algebra H R , j (Ts ) = −qs Ts−1 = (qs − 1)T1 − Ts , (b)

for all s ∈ 5 and j (Tσ ) = Tσ for all σ ∈ . This automorphism exchanges x J and y J modulo a unit j (x J ) = c J y J , where c J ∈ R is invertible, for J ⊂ 5.

Proof (a) j (Ts ) satisfies the equation (X + 1)(X − q) = 0. We assume that j is an automorphism (see [IM, 3.2], [DJ1, §2]). (b) See [DJ1, p. 25]. 4.2.10 LEMMA

H R is a free left (resp., right) H R,J module with basis (Td )d∈D J,∅ (G) (resp., (Td )d∈D∅,J (G) ).

Proof We have Waff (G) = W J D J,∅ (G) = D∅,J (G)W J with addition of the lengths by Section 4.1.1(a). 4.2.11 LEMMA

L Let d ∈ D J,K (G). The R-submodule R(I \PJ d PK /I ) ' w∈W J dW K RTw of H R is a free right H R,J -module (resp., left H R,K -module) M M M RTw = Ta Td H R,K = H R,J Td Tb w∈W J dW K

J a∈D∅,K ∩dJ

b∈D K−1 d

K ∩ J,∅

J K with basis Ta Td for a ∈ D∅,K ∩ d J (resp., Td Tb for b ∈ Dd −1 K ∩ J,∅ ).

Proof J K We have I \PJ d PK /I ' W J d W K = D∅,K ∩ d J dW K = W J d Dd −1 K ∩ J,∅ with addition of the lengths for the last two expressions by Section 4.1.1(a).

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

57

4.2.12 Proof of Theorem 4.2.1 Let V ∈ Mod H R,J . We want to describe V ⊗H R,J H R as a right H R,K -module. We have a decomposition of the (H R,K , H R,J )-module M HR ' R[I \PJ d PK /I ]. d∈D J,K (G)

For d ∈ D J,K (G) we want to understand the H R,K -module V ⊗H R,J R[I \PJ d PK /I ]. We apply Lemma 4.2.11: V ⊗H R,J R[I \PJ d PK /I ] ' R V ⊗H R,J



 Ta Td H R,K .

M J a∈D∅,d K ∩J

P J We have a V Ta = V because 1 ∈ D∅,d K ∩J , and we can forget the a in the righthand side. We have an isomorphism of the H R,K -module F Kd ,J (V ) ' (V ⊗ Td ) ⊗H R,K ∩d −1 J H R,K .

4.2.13 We deduce from Theorem 4.2.1 a basis for HomH R (x J H R , x K H R ). The right module x J H R is induced from the trivial or index character of H R,J ,  x J H R ' x J H R,J ⊗H R,J H R = h ∈ H R Tw h = qw h for all w ∈ W J . The proof is the same as in Lemma 4.2.7 using Lemma 4.2.10. The restriction is the right adjoint of the induction, and with Theorem 4.2.1 we get M HomH R (x J H R , x K H R ) = HomH R,J (x J H R,J , R[PK \PK d PJ /I ]). d∈D K ,J (G)

Let aw ∈ R for w ∈ W K dW J . Then  X   Tv aw Tw Tu = qw qu w∈W K dW J

X

 aw Tw ,

w∈W K d W J

for all v ∈ W K and u ∈ W J , is equivalent to aw constant for all w ∈ W K d W J . (The proof is the same as in Lemma 4.2.7.) We deduce the following. BASIS

HomH R (x J H R , x K H R ) has a basis (8dK ,J ) parametrized by D K ,J (G), X 8dK ,J x J = Tw . w∈W K d W J

´ MARIE-FRANCE VIGN ERAS

58

In particular, We have the following: • 81∅,J : x J H R → H R is the inclusion; • 81J,∅ : H R → x J H R is the left multiplication by x J ; • 8w ∅,∅ = Tw for all w ∈ Waff (G). 4.2.14 Proof of Theorem 2.3 As a right H R -module, R[PJ \G/I ] = x J H R . From Sections 4.1.6 and 4.2.13, the functor of I -invariants gives an isomorphism Hom RG R[PJ \G], R[PK \G]) ' HomH R (x J H R , x K H R ) sending 9 Kd ,J on 8dK ,J for all d ∈ D K ,J (G). Hence the Schur algebra S R of G is also the ring of endomorphisms of an Rrepresentation of G, S R = EndH R ⊕ J x J H R ' End RG ⊕ J R[PJ \G].

4.2.15. The Schur algebra, basis, relations, generators From Section 4.2.13, a basis of the Schur algebra S R is (8dK ,J ), where K , J are proper subsets of 5 and d ∈ D K ,J (G), X 8dK ,J x J 0 = δ J,J 0 Tw , w∈W K d W J

and δ J,J 0 = 1 if J =

J0

and zero otherwise. The product 0

8dK ,J 8dK 0 ,J 0 = 0

(0) 0

of two elements of the basis is zero when J 6= K 0 . The product 8dK ,L 8dL ,J is a finite P 00 sum ad 00 8dK ,J . The basis elements 81∅,J , 81J,∅ , Tw satisfy the following relations: (a) Ts 81∅,J = qs 81∅,J for s ∈ J, (b) 81J,∅ Ts = qs 81J,∅ for s ∈ J, (c) 81∅,J 81J,∅ = x J , (d) 81K ,∅ Td 81∅,J = Pd J ∩ K (q)8dK ,J . They are all evident; (d) is equivalent to the basic formula in Lemma 4.2.8, x K Td x J = PK ∩ d J (q)x J . We get a presentation of the Schur algebra S R by generators 81∅,J , 81J,∅ , Ts , Tσ for J a proper subset of 5, s ∈ 5, σ ∈ , by relations (0), (a), (b), (c), (d), and by the relations in H R when PJ (q) is invertible in R for all J ⊂ 5.

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

59

4.3. Going between groups and Hecke algebras Let R be equal to any commutative ring. The functor of invariants by the Iwahori subgroup I or the minimal parabolic subgroup B J , inv I : Mod R G → Mod H R ,

inv B J : Mod R M J → Mod H R,J ,

relates the R-representations of G or M J with the right modules for the IwahoriHecke algebra H R or the finite Hecke algebra H R,J . We show that the parahoric induction (or restriction) on the group side corresponds via inv I , inv B J to the analogous functors on the Hecke algebra side. 4.3.1 THEOREM

The functors iH J ◦ inv B J : Mod R M J → Mod H R,J → Mod H R , inv I ◦ i JG : Mod R M J → Mod R G → Mod H R are equal. Proof Let V ∈ Mod R M J . By Theorem 4.1.4, inv I ◦ i JG (V ) is (i JG V ) I =

M

VdI ,

(4.3.2)

d∈D J,∅ (G)

where VdI is the R-module of functions of i JG (V ) with support PJ d I and right invariant by I , and the value at d gives an isomorphism VdI ' R V B J . We denote by f v,d the function in VdI with value v ∈ V B J at d. As H R is a free left H R,J -module of basis (Td ) for d ∈ D J,∅ (G) by Lemma 4.2.10, i H J ◦ inv B J (V ) is V B J ⊗H J H R =

M

V B J ⊗ Td .

(4.3.3)

d∈D J,∅ (G)

We compare now the right actions of H R on (4.3.2) and (4.3.3). Let w ∈ Waff (G), and let f = f v,1 : PJ → R be a right I -invariant function. By definition, X [ f Tw (−) = f (−g −1 ) if I w I = I g (disjoint union). g

g

´ MARIE-FRANCE VIGN ERAS

60

The support of f Tw is PJ d I . When w ∈ W J , then Tw ∈ H R,J = H R (PJ , I ). We have f (1) ∈ V B J and f Tw (1) = f (1)Tw . When d ∈ D J,∅ (G), we have f Td (d) = f (1) S because the terms in g f (dg −1 ), I d I = g I g vanish if I g 6= I d. Indeed, g = d x, x ∈ I, and d x −1 d −1 = y ∈ PJ implies Td = Ty Td by additivity of the lengths (Section 4.1.1(a)); hence y ∈ I . Let φ ∈ H R . We write φ = hTd for h ∈ H R,J and d ∈ D J,∅ (G). We clearly have (v ⊗ T1 )hTd = vh ⊗ Td and, by the above computation, f v,1 hTd = f vh,1 Td = f vh,d . We deduce that the map P

v ⊗ Td → f v Td : V B J ⊗ Td → VdI is an isomorphism of right H R -modules. 4.3.4 THEOREM

The functors inv B J ◦ r JG : Mod R G → Mod R PJ → Mod H R,J and r JH ◦ inv I : Mod R G → Mod H R → Mod H R,J are equal. Proof U J is a normal subgroup of I with quotient I /U J = B J . 4.3.5 COROLLARY

Let d ∈ D K ,J (G). Then the functors F Kd ,J ◦ inv B J : Mod R M J → Mod H R,J → Mod H R,K

and inv B K ◦ FKd ,J : Mod R M J → Mod R M K → Mod H R,K

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

61

are equal. The functors TKH,J ◦ inv B J : Mod R M J → Mod H R,J → Mod H R,K , inv B K ◦ TKG,J : Mod R M J → Mod R M K → Mod H R,K are equal. Our motivation is to study the homomorphism of I -invariants when we restrict to parahorically induced representations. The above results allow us to reduce from the p-adic case to the finite case. Let V J ∈ Mod R M J , VK ∈ Mod R M K for two proper subsets J, K of 5. 4.3.6 PROPOSITION

The homomorphism of the I -invariants I Hom RG (i KG VK , i JG V J ) → HomH R (i KH VK ) I , (i H J VJ )




is injective (resp., surjective, bijective) if and only if the homomorphisms of B K invariants
Hom R M K (VK , FKd ,J V J ) → HomH R,K VKB K , (FKd ,J V J ) B K
' HomH R,K VKB K , F Kd ,J (V JB J ) are injective (resp., surjective, bijective) for all d ∈ D K ,J (G). Proof We have, by adjunction and decomposition of TKG,J , M Hom RG (i KG VK , i JG V J ) ' Hom R M K (VK , FKd ,J V J ).

(a)

d∈D K ,J (G)

Using the fact that the parahoric induction commutes with the invariants, adjunction, and decomposition of TKH,J and, finally, the fact that the parahoric restrictioninduction commutes with the invariants, one gets M

HomH R (i KG VK ) I , (i JG V J ) I ' HomH R,K VKB K , (FKd ,J V J ) B K . (b) d∈D K ,J (G)

It is easily seen that the homomorphism of the I -invariants from (a) to (b) on the left side respects the decomposition and corresponds to the homomorphism of the B K -invariants on each term of the right side. We apply Corollary 4.3.5 to replace (FKd ,J V J ) B K by F Kd ,J (V JB J ).

´ MARIE-FRANCE VIGN ERAS

62

4.3.7 Remark When VK is generated by its B K -invariants, i KG VK is generated by its I -invariants and the homomorphism of I -invariants in Proposition 4.3.6 is injective. If, moreover, R is a field of characteristic not equal to p, VK , V J are finite-dimensional and the equality of the dimensions
dim Hom R M K (VK , FKd ,J V J ) = dim HomH R,K VKB K , F Kd ,J (V JB J ) for all d ∈ D K ,J (G) is equivalent to the bijectivity of the homomorphisms of I invariants in Proposition 4.3.6. 5. Proof of the main theorem We now prove the main theorem from Section 1.2. As most of the arguments are valid for a general reductive group, we restrict to GL(n, F) only at the end. 5.1 Let G be a general reductive p-adic group as in Sections 2.1 and 4.1. The group of Fq -points of the maximal split torus T of G is a maximal split torus of M J , and the set 8 J of roots of M J with respect to this torus are described in [Ti, 3.5.1]. We denote by 8+ J , 1 J ' J, the set of positive, simple roots of 8 J with respect to the minimal parabolic subgroup B J of M J . The order of the unipotent subgroup Uα of M J attached to α ∈ 1 J is q ds , where s is the reflection associated to a simple affine root in J identified with α. One considers the following subset U∅∗,J of the unipotent radical U∅J of B J : Y U∅∗,J := Uα . (5.1) α∈8+ J −1 J

The commutator relations show that U∅∗,J is a normal subgroup of U∅J and that the quotient U∅J /U∅∗,J is abelian. When K ⊂ J , we have the exact sequence (Section 4.1.1) 1 → U J → U K → U JK → 1 (5.2) K ⊂J and U JK =

Y

Uα ,

+ α∈8+ J −8 K

U∅∗,J ⊂ U∅∗,K U JK =

Y α∈8+ J −1 K

Uα .

(5.2)0K ⊂J

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

63

We have U∅J = U∅K U JK . The group U∅ is the pro- p-radical of the Iwahori subgroup I = P∅ . The group U∅∗,J lifts to an open compact subgroup V∅∗,J of U∅ using the exact sequence (5.2)∅⊂J . LEMMA

Suppose K ⊂ J . Then V∅∗,J ⊂ V∅∗,K . Proof The inverse image by (5.2)∅,J of the inclusion (5.2)0K ⊂J is the inclusion of the lemma.

5.2 Let G be the group of rational points of a reductive connected group over the finite field Fq . We use the same notation as above (with G = M J ). T is a maximal split torus; W is the Weyl group; B(= B J ) = ZU is a minimal parabolic subgroup with unipotent radical U (= U∅J ) and Levi subgroup a torus Z ; 8 are the roots of (G, T ); 1 are the simple roots U ∗ (= U∅∗,J ). A parabolic subgroup P = M V with unipotent radical V and Levi M is standard if B ⊂ P, Z ⊂ M. Definition Let R be any commutative ring. An R-character χ : U → R ∗ of U is called nondegenerate if χ is not trivial on Uα for any simple root α ∈ 1 and called generic if χ is nondegenerate and trivial on U ∗ . The R-representation of G, G 0 R,χ := indU χ

induced by a generic R-character χ of U is called a Gelfand-Graev R-representation of G. When the generic characters are conjugate in G, there is a unique Gelfand-Graev Rrepresentation of G modulo isomorphism. When G is a torus Z , then U = {1} and the (unique) Gelfand Graev representation of Z is the regular representation 0 R = i 1Z 1 ' R[Z ]. Note that zero is the only G-invariant element of 0 R,χ when G is not a torus. If G = Z is a torus, then 0 RZ ' R. Let ` be a prime number different from p. We decompose Z = Z ` Z ` where Z ` is of order a power of ` and Z ` is of order prime to `. Set B = Z ` B ` with B ` = Z ` U (the prime-to-` part of B). When ` does not divide |Z |, we have Z ` = {1} and B = B ` .

´ MARIE-FRANCE VIGN ERAS

64

We collect facts on a Gelfand-Graev representation that is used later. We insist on keeping a commutative ring R as general as possible and a general reductive connected finite group G. 5.3 LEMMA

Let R be a commutative ring such that U has a generic R-character χ . Then the Gelfand-Graev R-representation 0 R,χ has the following properties. (1) The U -invariants of 0 R,χ are 0U R,χ ' R[Z ]. (2) (3)

We have Hom RG (indG 1, 0 R,χ ) ' R[Z ` ]. B` The B-invariants of 0 R,χ are isomorphic to the sign character as a right H R (G, B)-module B 0 R,χ ' sign .

(4)

Let P = M V be a standard parabolic subgroup of G. The V -invariants of 0 R,χ are a Gelfand-Graev representation of M, 0 VR,χ ' 0 R,w Mo wo (χ )| M , where wo , respectively, w Mo , is the longest element of W , respectively, W M , and w Mo wo (χ)| M : U ∩ M → R ∗ is the generic character u 7→ χ (wo w Mo uw Mo wo ). The P-invariants of 0 R,χ are zero if P 6= B.

Proof Note that (1) is a particular case of (4). We prove (4). The objects relative to M are denoted with an index M. By the Mackey decomposition (Theorem 4.1.4), 0 VR,χ is a direct sum indexed by W/W M . Let w ∈ W/W M be distinguished. We have Y wV w−1 ∩ U = Uw(α) . + α∈8+ −8+ M , w(α)∈8

−1 This group contains no Uβ with β ∈ 1 if and only if w−1 (β) ∈ 8+ M or w (β) < 0 for any β ∈ 1. This is equivalent to w ∈ wo w Mo (see [C, p. 262]). One deduces by Theorem 4.1.4 that 0 VR,χ is the set of functions f : U wo w Mo M V → R such that f (uwo w Mo mv) = χ (u) f (wo w Mo m) for all u ∈ U, m ∈ M, v ∈ V with the natural action of M. By restriction to M we deduce the first part of (4). The second part of M (4) comes from the fact that 0 R,w = 0 if M is not a torus. Mo wo (χ)| M We prove (2) using adjunction and (1). We get Hom RG (indG 1, indUM χ ) ' B` `

R[Z ` Z ` ] Z ' R[Z ` ].

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

65

B is a free R-module of rank 1, the set of functions We prove (3). From (1), 0 R,χ f : U wo B → R such that f (uwo b) = χ(u) f (wo ) for all u ∈ U, b ∈ B. Let s ∈ W be a reflection associated to a simple root α ∈ 1. Then B ∪ Bs B is a standard B∪Bs B = 0 by (4). Hence f [B] + f [Bs B] = 0 for any parabolic subgroup and 0 R,χ B B f ∈ 0 R,χ . Hence 0 R,χ is the sign representation.

When R is a field of characteristic not equal to p, the representation indG B 1 is quasi projective (see [V3, Proposition I.3]) and the B-invariant functor V → V B : Mod R (G) → Mod R H R (G, B) induces a bijection between the irreducible representations of G generated by their B-invariant vectors and the simple modules for H R (G, B). 5.4 Definition When R is a field of characteristic not equal to p, we denote by St R the unique irreducible representation of G corresponding to the character sign of H R (G, B) by the B-invariant functor. By Lemma 5.1(1), we have an isomorphism of locally profinite groups Y U/U ∗ ' Uα . α∈1

An R-character χ of U trivial on U ∗ defines a standard parabolic subgroup P = M V containing B, associated to the simple roots α ∈ 1 such that χ is not trivial on Uα . The character χ is the inflation to U of a generic character of U M = U ∩ M. In this way the R-characters of U which are trivial on U ∗ are in bijection with the disjoint union of the generic characters of U M for all standard M. We go back to the p-adic case. We use the notation introduced in Section 5.1, Definition 5.2, and Lemma 5.3, which introduces an index J for objects associated to the finite reductive group M J . Let R be any commutative ring where p is invertible and such that the characteristic function of the finite group Uα is the sum of the Rcharacters of Uα , for all simple roots α ∈ 5. We introduce the R-representation of G, M M ∗ 0 R := R[U5−i \G] = ei H R (G), (5.3) i∈5

i∈5

∗ ∗ where U5−i := V∅∗,5−i and ei is the idempotent of H R (G) associated to U5−i for all i ∈ 5. The representation 0 R ∈ Mod R G is projective and finitely generated. Recall that J always denotes a proper subset of 5.

´ MARIE-FRANCE VIGN ERAS

66

5.5 PROPOSITION

(1)

The R-representation 0 R of G is isomorphic to 0R '

M |5|−|J M|M χ ∈Y J

J

(2)

i JG 0 R,χ,J ,

where Y J is the set of generic R-characters of the unipotent radical of B J and 0 R,χ,J is the Gelfand-Graev R-representation of M J associated to χ ∈ Y J . The I -invariants of 0 R are an H R -module isomorphic to 0 RI

'

M |Y J |(|5|−|J M |)

yJ HR ,

J

(3)

where H R,J y J = sign J is the sign character of H R,J (see (4.2.4)). The ring of H R -endomorphisms of 0 RI is Morita equivalent to the Schur algebra S R , EndH R (0 RI ) 'Morita S R .

Proof We prove property (1) of the proposition. Let us fix i ∈ 5. Our hypothesis on R allows us to write ei as an orthogonal sum of idempotents eχ for all R-characters χ of U∅5−i trivial on U∅5−i,∗ . Such a character χ is the inflation of a generic character of the unipotent radical U∅J of B J for some J ⊂ 5 − i. We identify the characters of the unipotent radical U∅J of B J with characters of U∅ trivial on U J via the exact sequence (5.2)∅⊂J in Section 5.1. The idempotent eχ identifies with an idempotent of the group R-algebra of M J , R[M J ] = H R (PJ , U J ) ⊂ H R (G). We have M M

eχ ,

(5.4)

eχ R[M J ] ' 0 R,χ ,J ;

(5.4)0

eχ H R (G) = i JG 0 R,χ ,J ,

(5.4)00

ei =

J ⊂5−i χ∈Y J

hence from which we deduce Proposition 5.5(1). We prove property (2) of the proposition. By Lemma 5.3, we have in Mod H R,J , BJ 0 R,χ ,J ' y J H R,J ' sign J .

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

67

Hence, by Theorem 4.3.1, we have in Mod H R , (i JG 0 R,χ ,J ) I ' y J H R .

(5.5)

We prove property (3) of the proposition. The algebra of H R -endomorphisms L L of J ⊂5 m J y J H R is Morita equivalent to the algebra of H R -endomorphisms of L L y J H R because all the m J are greater than or equal to 1 and we can replace J ⊂5 L L them by 1 by Claim 3.6. The algebra of H R -endomorphisms of J ⊂5 y J H R is isomorphic to the Schur algebra MM S R := EndH R x J HR J

because H R has an automorphism of order 2 permuting x J , y J modulo a unit in R for all J by Claim 4.2.9. We introduce the property Ho for the Gelfand-Graev representations. Property Ho The B J -invariant homomorphism Hom R M J (0 R,χ ,J , 0 R,χ 0 ,J ) → EndH R,J sign J

(5.6) J

is surjective for all generic characters χ, χ 0 ∈ Y J for all J . The surjectivity (5.6) J is clearly satisfied in the usual case, where the Gelfand-Graev representations of M J are isomorphic. 5.6 LEMMA

The I -invariant algebra homomorphism End RG 0 R → EndH R (0 RI )

(5.6)

is surjective when property Ho is true. In particular, (5.6) is surjective when G = GL(n, D), where D is a division algebra over F. Proof By (5.4) and (5.5), we have to prove that the homomorphism of I -invariants
HomH R (G) eχ H R (G), eµ H R (G) → HomH R (y K H R , y J H R )

´ MARIE-FRANCE VIGN ERAS

68

is surjective for any pair of characters (χ, µ) ∈ Y K × Y J and for all proper subsets K , J of 5. By Proposition 4.3.6, we have to prove that Hom R M K (0 R,χ,K , FKd ,J 0 R,µ,J ) → HomH R,K (sign K , F Kd ,J sign J ) is surjective for all d ∈ D K ,J (G). By definition, FKd ,J = i KK ∩ d J ◦ Int(d) ◦ rdJ−1 K ∩J with a similar formula for F Kd ,J . The image of a Gelfand-Graev representation by parahoric restriction is a GelfandGraev representation by Lemma 5.3(4); the same is true for the conjugate by Int(d) and on the Hecke algebra side for the sign. The surjectivity is reduced to the surjectivity of (5.6) J for all J . 5.7 Let J R be the annihilator of R[I \G] in the global Hecke algebra H R (G). Set J R∗ := { f ∗ | f ∈ J R }, where f ∗ (g) := f (g −1 ) for all g ∈ G. An RG-endomorphism L of 0 R = i∈5 ei H R (G) is given by left multiplication by a matrix (a ji ), where a ji ∈ e j H R (G)ei , M MX MX ei xi → a ji ei xi = a ji xi , (5.7) i∈5

j∈5 i∈5

j∈5 i∈5

for xi ∈ H R (G). LEMMA

The kernel of the I -invariants algebra homomorphism End RG 0 R → EndH R (0 RI ) is the set of matrices (a ji ) with a ji ∈ e j J R∗ ei for all i, j ∈ 5. Proof The RG-endomorphism of 0 R given by (a ji ) has a zero restriction to 0 RI = L i∈5 ei R[G/I ] if and only if a ji R[G/I ] = 0 for all i, j ∈ 5. The map f → ∗ f is an isomorphism from H R (G) to its opposite algebra H R (G)o and sends R[G/I ] to R[I \G]. Hence a ji R[G/I ] = 0 is equivalent to a ji ∈ J R∗ . We have J R∗ ∩ e j H R (G)ei = e j J R∗ ei . We deduce from Lemma 5.6 that EndH R (0 RI ) is isomorphic to the algebra of matrices (a ji ) with a ji ∈ e j (H R (G)/J R∗ )ei for all i, j ∈ 5. It is clear that ei is not zero in the quotient H R (G)/J R∗ because ei R[G/I ] 6= 0 and we do not change the notation for its image.

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

69

5.8 We introduce the R-representation of G, Q R := 0 R / 0 R J R =

M

ei (H R (G)/J R ).

(5.8)

i∈5

It is clear that Q R is finitely generated and projective in Mod H R (G)/J R . We have End RG Q R ' End RG 0 R / Hom RG (0 R , 0 R J R )

(5.9)

because the kernel 0 R J R of the surjective homomorphism 0 R → Q R is stable by End RG 0 R . Hence the ring End RG Q R is isomorphic to the algebra of matrices (a ji ) with a ji ∈ e j (H R (G)/J R )ei for all i, j ∈ 5. We deduce the following from Proposition 5.5 and Lemmas 5.6 and 5.7. PROPOSITION

When the property Ho is true, we have a Morita isomorphism (End RG Q R )o 'Morita S R . 5.9 From Claim 3.7, the functor HomH R (G) (Q R , −) : Mod H R (G)/J R → Mod(End RG Q R )o

(5.10)

is an equivalence of categories, and Q R is a progenerator of Mod H R (G)/J R , if and only if Hom RG (Q R , V ) 6= 0 for any V ∈ Irr R G annihilated by J R . By (5.3) and ∗ (5.8), Hom RG (Q R , V ) 6= 0 if and only if V has a nonzero vector invariant by U5−i for some i ∈ 5. 5.10 We suppose, from now until the end of this section, that R is an algebraically closed field of characteristic not equal to p in order to have references for the following properties. (a) The R-representations of level zero of G form a sum of blocks, and indG I 1 R is of level zero (see [V1]). Hence the R-representations of G annihilated by J R are of level zero. (b) For GL(n, F) the irreducible R-representations annihilated by J R are irreducible unipotent representations (subquotients of indG I 1 R ) (see [V3]). (c) For each irreducible R-representation V of level zero of G, there exists a proper subset J of 5 and an irreducible cuspidal R-representation σ J of M J such that V is a quotient of i JG σ J ; (5.11)

´ MARIE-FRANCE VIGN ERAS

70

that is, the inflation to PJ of σ J is contained in the restriction to PJ of V (see [V6]). We say that σ J is G-relevant when i JG σ J has an irreducible quotient annihilated by J R . (d) For GL(n, F) the irreducible cuspidal G-relevant R-representations of M J are J the irreducible cuspidal subquotients of ind M BJ 1R . We consider the following property (true when G = GL(n, F)). Property H1 For any proper subset J ⊂ 5, any irreducible cuspidal G-relevant R-representation σ J of M J has a nonzero vector invariant by U∅∗,J (see Section 4.1.1). H1 is trivially true when there are no irreducible cuspidal G-relevant R-representation σ J of M J for all proper subsets J of 5. PROPOSITION

We have Mod H R (G)/J R ' Mod S R and Q R is a progenerator of Mod H R (G)/J R if the properties Ho , H1 are true. Proof Let V ∈ Irr R G be annihilated by J R . Then V is of level zero and contains a relevant type σ J . There exists i ∈ 5 such that J ⊂ 5 − i. By H1 , V has a nonzero vector invariant by V∅∗,J , and by Lemma 5.1, V∅∗,J contains V∅∗,5−i . By definition (5.3), ∗ U5−i = V∅∗,5−i ; hence Hom RG (Q R , V ) 6= 0. We apply Section 5.9. 5.11. Remarks on Property H1 (a) H1 is not true in general. Property H1 means that a cuspidal irreducible G-relevant representation σ J is generic, that is, contains a generic character of U∅J (the unipotent radical of B J ). Let G = GU3 (Fq ). Over Q` , by [G], the group G has three irreducible unipotent (in the sense of G. Lusztig) representations: trivial 1, Steinberg St, and cuspidal π. Let ` 6= 2, 3, p, and let ` divide q + 1. Then r` π is irreducible and r` St = 1 + αr` π + ρ,

(5.12)

where ρ ∈ IrrF` G, 2 ≤ α ≤ (`d + 1)/3, and `d is the highest power of ` dividing q + 1. The representation ρ is irreducible cuspidal and generic, and r` π is irreducible cuspidal and not generic. But r` π is relevant. (b) The irreducible cuspidal R-representations of a finite linear group are generic (see [V1]). We say that we are in the linear case when the groups M J are isomorphic Q to product of finite linear groups i GL(n i , Fq ). In the linear case, Properties Ho and H1 are true. In particular, they are true when G = GL(n, D) where D/F is a division algebra of finite dimension (D = F in particular).

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

71

5.12 The abelian subcategory of Mod R G generated by V ∈ Mod R G is the full subcategory of R-representations of G with irreducible subquotients isomorphic to subquotients of V . We introduce the abelian subcategory B R,1 (G) of Mod R G generated by R[I \G]. It is the unipotent block when G = GL(n, F) for all R, or when G is general for R = C. We call the representations of B R,1 (G) unipotent. When G is replaced by a finite reductive group, we give the same definition with a minimal parabolic subgroup B instead of I . A type of level zero in an irreducible unipotent R-representation is relevant (Section 5.9). We consider the properties H2 and H3 (true when G = GL(n, F)). Property H2 B R,1 (G) is a direct factor of Mod R (G). Then we can define the unipotent part V1 of V ∈ Mod R G, for instance, 0 R,1 . Property H2,J B R,1 (M J ) is a direct factor of Mod R (M J ) for all J . Then we can define the unipotent part V1 of V ∈ Mod R M J , for instance, 0 R,χ J ,1 for χ ∈ Y J . Let N be the integer equal to the maximum length of 0 R,χ J ,1 for all J and all χ ∈ Y J . Recall that J R is the annihilator of R[I \G] in the global Hecke algebra H R (G). 5.13 THEOREM

(1) (2)

When properties H1 and H2 are true, the unipotent part 0 R,1 of 0 R is a progenerator of B R,1 (G). When properties H2 and H2,J are true, 0 R,1 =

M |5|−|J M |M J ⊂5

(3)

χJ

i JG 0 R,χ J ,1 .

(5.13)

When properties H1 , H2 , and H2,J are true, J RN annihilates the unipotent block B R,1 (G).

Proof The proof of property (1) is as in the proof of Proposition 5.10. The unipotent representation 0 R,1 is finitely generated and projective by (3). Any simple unipotent Rrepresentation V of G contains a relevant type of level zero, and Property H1 implies Hom RG (0 R,1 , V ) 6= 0. We apply Claim 3.7.

´ MARIE-FRANCE VIGN ERAS

72

Property (2) results from Lemma 5.14. We prove property (3), admitting (2). Each i JG 0 R,χ J ,1 has a finite filtration of length less than or equal to N with quotients of the form i JG ρ, where ρ is an irreducible subquotient of R[B J \M J ]. Hence i JG ρ is isomorphic to a subquotient of i ∅G 1 = R[I \G] and is annihilated by J R . We deduce that J RN annihilates i JG 0 R,χ J ,1 for all J, χ J . By property (2), J RN annihilates 0 R,1 . By property (1), J RN annihilates B R,1 (G). 5.14 We show that the property of being unipotent is compatible with the functors of parahoric induction or restriction. Once this is proved, we get part (2) of Lemma 5.13. The strong compatibility is related to a weak form H3 of the conjecture of the unicity of the supercuspidal support, known for G = GL(n, F) (see [V3]). We recall that a representation π ∈ Irr R G is supercuspidal when it is not a subquotient of a proper parabolically induced representation. The same definition is given in the finite case. Property H3 For any irreducible supercuspidal σ ∈ Irr R M J with (J, σ ) different from (∅, 1), the representations i JG σ and R[I \G] have no isomorphic irreducible subquotients. LEMMA

Let π (a1 ) (a2 ) Let π (b1 ) (b2 )

∈ Irr R M J . Then i JG π is unipotent if π is unipotent; if π is not unipotent, no subquotient of i JG π is unipotent if H3 is true. ∈ Irr R G. Then r JG π is unipotent if π is unipotent; if π is not unipotent, no nonzero subquotient of r JG π is unipotent if property H3 is true.

As usual, the proof for the p-adic case is valid for the finite case. Proof J G J (a1 ) Let π ∈ Irr R M J . If π is a subquotient of ind M B J 1 = i ∅ 1, then i J π is a subquotient of i JG i ∅J 1 = i ∅G 1. (5.14) (b1 ) Let π ∈ Irr R G. If π is a subquotient of R[I \G] = i ∅G 1, then r JG π is a subquotient of (Theorem 4.1.4) M r JG i ∅J 1 = i ∅J 1. (5.15) w∈Waff (G)/W J

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

73

(a2 ) Let π ∈ Irr R M J be not isomorphic to a subquotient of i ∅J 1. There exist K ⊂ J and σ ∈ Irr R M K supercuspidal with (K , σ ) 6= (∅, 1) such that π is isomorphic to a subquotient of i KJ σ . Any subquotient of i JG π is a subquotient of i KG σ by transitivity of the induction. If property H3 is true for σ , then i JG π and R[I \G] have no isomorphic irreducible subquotients. (b2 ) Let π ∈ Irr R G be not isomorphic to a subquotient of i ∅G 1, and let J be a proper subset of 5 such that r JG π 6= 0. If π is not of level zero, then r JG π has no subquotient of level zero and hence no unipotent subquotient. We may suppose π to be of level zero. There exist K ⊂ J and σ ∈ Irr R M K supercuspidal with (K , σ ) 6= (∅, 1) such that π is isomorphic to a subquotient of i KG σ . Hence r JG π is a subquotient of (Theorem 4.1.4) M (5.16) i KJ Int w · σ ; r JG i KG σ = w∈W K \Waff (G)/W J , wK =K

σ is not a subquotient of i ∅K 1 because π is isomorphic to a subquotient of i KG σ but not of i ∅G 1 and (14). The same is true for Int wσ because Int w permutes the subquotients of i ∅K 1. We apply (a2 ) to M J instead of G to finish the proof. This ends the proof of the main theorem. Acknowledgments. This work is an answer to a question raised by Frenkel and was done during my stay at the Mathematical Sciences Research Institute in the fall of 2000, thanks to a delegation at the Centre National de la Recherche Scientifique. I thank J. Rogawski, D. Blasius, D. Ramakrishnan, R. Boltje, E. Baruch, G. Bhowmik, B. Mazur, and R. Taylor for their invitations to give a talk on these results at the University of California at Los Angeles, the California Institute of Technology, the University of California Santa Cruz, l’Universit´e des Sciences et Technologies de Lille, and Harvard University. Special thanks to Edward Frenkel for introducing me to the subject and to Raphael Rouquier for indicating to me the reference to M. Takeuchi and for interesting discussions. The final draft of this paper was done at the Institute for Advanced Study at Princeton in the first term of 2001. I am thankful for this invitation. References [B]

A. BOREL, Admissible representations of a semi-simple group over a local field with

[Bo]

vectors fixed under an Iwahori subgroup, Invent. Math. 35 (1976), 233 – 259. MR 56:3196 53 ´ ements de math´ematique, fasc. 34: Groupes et alg`ebres de Lie, N. BOURBAKI, El´ chapitres 4 – 6, Actualit´es Sci. Indust. 1337, Hermann, Paris, 1968. MR 39:1590 53

74

[BT1] [BT2] [C]

[CPS] [CR1]

[CR2]

[D] [Di]

[DD] [DJ1] [DJ2] [G]

[Gr] [Gre] [IM]

[M1] [M2] [P]

´ MARIE-FRANCE VIGN ERAS

´ F. BRUHAT and J. TITS, Groupes r´eductifs sur un corps local, Inst. Hautes Etudes Sci. Publ. Math. 41 (1972), 5 – 251. MR 48:6265 42, 48, 49 , Groupes r´eductifs sur un corps local, II: Sch´emas en groupes, Inst. Hautes ´ Etudes Sci. Publ. Math. 60 (1984), 197 – 376. MR 86c:20042 42, 48, 49 R. W. CARTER, Finite Groups of Lie Type: Conjugacy Classes and Complex Characters, Pure Appl. Math., Wiley-Interscience, Wiley, New York, 1985. MR 87d:20060 48, 49, 54, 64 E. CLINE, B. PARSHALL, and L. SCOTT, Generic and q-rational representation theory, Publ. Res. Inst. Math. Sci. 35 (1999), 31 – 90. MR 2000i:20021 39 C. CURTIS and I. REINER, Methods of Representation Theory: With Applications to Finite Groups and Orders, Vol. I, Pure Appl. Math., Wiley-Interscience, Wiley, New York, 1981. MR 82i:20001 , Methods of Representation Theory: With Applications to Finite Groups and Orders, Vol. II, Pure Appl. Math., Wiley-Interscience, Wiley, New York, 1987. MR 88f:20002 J.-F. DAT, Types et inductions pour les repr´esentations modulaires des groupes ´ p-adiques, Ann. Sci. Ecole Norm. Sup. (4) 32 (1999), 1 – 38. MR 99m:22018 40 R. DIPPER, “Polynomial representations of finite general linear groups in nondescribing characteristic” in Representation Theory of Finite Groups and Finite-Dimensional Algebras (Bielefeld, 1991), Progr. Math. 95, Birkh¨auser, Basel, 1991, 343 – 370. MR 92h:20018 R. DIPPER and S. DONKIN, Quantum GLn , Proc. London Math. Soc. (3) 63 (1991), 165 – 211. MR 92g:16055 R. DIPPER and G. JAMES, The q-Schur algebra, Proc. London Math. Soc. (3) 59 (1989), 23 – 50. MR 90g:16026 39, 56 , q-tensor spaces and q-Weyl modules, Trans. Amer. Math. Soc. 327 (1991), 251 – 282. MR 91m:20061 39 M. GECK, Irreducible Brauer characters of the 3-dimensional special unitary groups in nondefining characteristic, Comm. Algebra 18 (1990), 563 – 584. MR 91b:20016 70 J. A. GREEN, Polynomial Representations of GLn , Lecture Notes in Math. 830, Springer, Berlin, 1980. MR 83j:20003 R. M. GREEN, The affine q-Schur algebra, J. Algebra 215 (1999), 379 – 411. MR 2000b:20011 38 N. IWAHORI and H. MATSUMOTO, On some Bruhat decomposition and the structure of ´ the Hecke rings of p-adic Chevalley groups, Inst. Hautes Etudes Sci. Publ. Math. 25 (1965), 5 – 48. MR 32:2486 53, 56 L. MORRIS, Tamely ramified intertwining algebras, Invent. Math. 114 (1993), 1 – 54. MR 94g:22035 42, 43, 48, 49 , Level zero G-types. Compositio Math. 118 (1999), 135 – 157. MR 2000g:22029 42 N. POPESCU, Abelian Categories with Applications to Rings and Modules, London Math. Soc. Monogr. 3, Academic Press, London, 1973. MR 49:5130 46, 47

SCHUR ALGEBRAS OF REDUCTIVE p-ADIC GROUPS, I

[SS]

[S]

[T] [Ti]

[V1] [V2]

[V3]

[V4]

[V5] [V6]

75

P. SCHNEIDER and U. STUHLER, Representation theory and sheaves on the Bruhat-Tits

´ building, Inst. Hautes Etudes Sci. Publ. Math. 85 (1997), 97 – 191. MR 98m:22023 49 I. SCHUR, “Uber die rationalen Darstellungen der allgemeinen linearen Gruppe (1927)” in Gesammelte Abhandlungen, III, Springer, Berlin, 1973, 68 – 85. MR 57:2858c M. TAKEUCHI, The group ring of GLn (q) and the q-Schur algebra, J. Math. Soc. Japan 48 (1996), 259 – 274. MR 98e:20016 39 J. TITS, “Reductive groups over local fields” in Automorphic Forms, Representations and L-Functions (Corvallis, Ore., 1977), I, Proc. Sympos. Pure Math. 33, Amer. Math. Soc., Providence, 1979, 29 – 69. MR 80h:20064 42, 43, 53, 62 ´ M.-F. VIGNERAS , Repr´esentations l-modulaires d’un groupe r´eductif p-adique avec ` 6= p, Progr. Math. 137, Birkh¨auser, Boston, 1996. MR 97g:22007 36, 69, 70 , “A propos d’une conjecture de Langlands modulaire” in Finite Reductive Groups (Luminy, France, 1994), Progr. Math. 141, Birkh¨auser, Boston, 1997, 415 – 452. MR 98b:22035 37 , Induced R-representations of p-adic reductive groups, with an appendix “Objets quasi-projectifs” by A. Arabia, Selecta Math. (N.S.) 4 (1998), 549 – 623. MR 99k:22026 37, 40, 45, 46, 65, 69, 72 , La conjecture de Langlands locale pour GL(n, F) modulo ` quand ´ ` 6= p, ` > n, Ann. Sci. Ecole Norm. Sup. (4) 34 (2001), 789 – 816. CMP 1 872 421 , Correspondance de Langlands semi-simple pour GL(n, F) modulo ` 6= p, Invent. Math. 144 (2001), 177 – 223. CMP 1 821 157 35 , “Irreducible modular representations of a reductive p-adic group and simple modules for Hecke algebras” in European Congress of Mathematics (Barcelona, 2000), I, Birkh¨auser, Basel, 2001, 117 – 133. CMP 1 905 316 50, 70

Institut de Math´ematiques de Jussieu, Unit´e Mixte de Recherche 7586, Case 247-4, place Jussieu 75252, Paris CEDEX, France; [email protected]

CONDITIONS FOR NONNEGATIVE CURVATURE ON VECTOR BUNDLES AND SPHERE BUNDLES KRISTOPHER TAPP

Abstract This paper addresses J. Cheeger and D. Gromoll’s question about which vector bundles admit a complete metric of nonnegative curvature, and it relates their question to the issue of which sphere bundles admit a metric of positive curvature. We show that any vector bundle that admits a metric of nonnegative curvature must admit a connection, a tensor, and a metric on the base space, which together satisfy a certain differential inequality. On the other hand, a slight sharpening of this condition is sufficient for the associated sphere bundle to admit a metric of positive curvature. Our results sharpen and generalize M. Strake and G. Walschap’s conditions under which a vector bundle admits a connection metric of nonnegative curvature. 1. Introduction A well-known question in Riemannian geometry is to what extent the converse of Cheeger and Gromoll’s soul theorem holds. Their theorem states that any complete noncompact manifold, M, with nonnegative sectional curvature is diffeomorphic to the normal bundle of a compact totally geodesic submanifold, 6 ⊂ M, called the soul of M (see [5]). The converse question is the classification problem: Which vector bundles over compact nonnegatively curved base spaces can admit complete metrics of nonnegative curvature? There are vector bundles that are known not to admit nonnegative curvature, but in all such examples the base space has an infinite fundamental group (see [13], [17], [1], [2]). Trivial positive results include all vector bundles over S 1 , S 2 , and S 3 , T S n for any n, and more generally all homogeneous vector bundles over homogeneous spaces. As for nontrivial positive results, D. Yang obtained nonnegatively curved metrics on rank 2 vector bundles over C P n #C P n (see [20]). More recently, K. Grove and W. Ziller constructed nonnegatively curved metrics on all vector bundles over S 4 and S 5 (see [7]). Our first result is a necessary condition for a vector bundle to admit a metric of nonnegative curvature. Suppose that M is an open (i.e., complete and noncompact) DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 1, Received 2 April 2001. Revision received 25 October 2001. 2000 Mathematics Subject Classification. Primary 53C20.

77

78

KRISTOPHER TAPP

manifold with nonnegative curvature, and suppose that 6 is a soul of M. Let ν(6) denote the normal bundle of 6, let p ∈ 6, let X, Y ∈ T p 6, and let W, V ∈ ν p (6). Let ∇ denote the connection in ν(6), and let R ∇ denote its curvature tensor, so that R ∇ (X, Y )W ∈ ν p (6). We can enlarge the domain of the tensor R ∇ by defining R ∇ (W, V )X to be the vector in T p 6 for which hR ∇ (W, V )X, Y i = hR ∇ (X, Y )W, V i for every Y ∈ T p 6. Let k6 describe the unnormalized sectional curvatures of 6; that is, let k6 (X, Y ) = hR(X, Y )Y, X i. Similarly, let k F (W, V ) = hR(W, V )V, W i describe unnormalized sectional curvatures of 2-planes perpendicular to 6. (F stands for fiber since k F really describes the curvatures of the fibers at points of 6.) By parallel transporting W and V along geodesics from p in 6, we can regard k F (W, V ) as a real-valued function on 6 near p; by hessk F (W,V ) (X ) we denote the hessian of this function in the direction X . We think of {R ∇ , k6 , k F } as the structure of M which is visible at points of the soul. Our necessary condition for nonnegative curvature is the following relationship between these visible structures. THEOREM A

If M is an open manifold of nonnegative curvature with soul 6, then for any p ∈ 6, X, Y ∈ T p 6, and W, V ∈ ν p (6),   2 h(D X R ∇ )(X, Y )W, V i2 ≤ |R ∇ (W, V )X |2 + hessk F (W,V ) (X ) · k6 (X, Y ). 3 An obvious question is whether the condition in Theorem A is sufficient; that is, if a vector bundle admits structures satisfying the inequality of Theorem A, then must it admit a metric of nonnegative curvature? We first discuss the case of connection metrics, about which much is already known. π A connection metric g E on the total space E of a vector bundle Rk → E → 6 is a metric arising from the following construction. Choose a Euclidean structure h·, ·i on the bundle (which means a smoothly varying choice of inner products on the fibers), a connection ∇ that is compatible with the Euclidean structure, a metric g6 on 6, and a rotationally symmetric metric g0 on Rk . Then there is a unique metric g E on E for which π : (E, g E ) → (6, g6 ) is a Riemannian submersion with horizontal distribution, H , determined by ∇, and with totally geodesic fibers isometric to (Rk , g0 ). By a connection metric g E 1 on the total space E 1 of the associated sphere π bundle S k−1 → E 1 → 6, we mean the intrinsic metric induced on the sphere of radius 1 about 6 in (E, g E ). In Theorem A, if the metric on M is a connection metric, then k F is parallel, so hessk F (W,V ) (X ) = 0. Therefore, the inequality becomes h(D X R ∇ )(X, Y )W, V i2 ≤ |R ∇ (W, V )X |2 · k6 (X, Y ).

(1.1)

This special case of Theorem A is not new. In [16], Strake and Walschap studied

CONDITIONS FOR NONNEGATIVE CURVATURE

79

conditions under which a vector bundle admits a connection metric of nonnegative curvature. Their necessary condition is stronger than that of inequality (1.1):   3 h(D X R ∇ )(X, Y )W, V i2 ≤ |R ∇ (W, V )X |2 · k6 (X, Y ) −  2 |R ∇ (X, Y )W |2 , 4 (1.2) where π > 0 is a bound on the diameters of spheres about the origin in (Rk , g0 ). We prove the following weak converse to Theorem A. THEOREM B

π

Let 6 be a compact manifold, let Rk → E → 6 be a vector bundle over 6, and let π S k−1 → E 1 → 6 be the associated sphere bundle. (1) E 1 admits a connection metric g E 1 of positive curvature if and only if the vector bundle admits a Euclidean structure h·, ·i, a compatible connection ∇, and a metric g6 on 6 such that the following inequality holds for all p ∈ 6, X, Y ∈ T p 6 with X ∧ Y 6= 0 , and W, V ∈ E p with W ∧ V 6= 0: h(D X R ∇ )(X, Y )W, V i2 < |R ∇ (W, V )X |2 · k6 (X, Y ). (2)

Further, if the vector bundle admits structures for which this inequality is satisfied, then E admits a complete connection metric g E of nonnegative curvature.

Notice that the strict inequality implies that (6, g6 ) has positive curvature. To prove Theorem B(2), we show that g0 can be chosen so that the connection metric g E on E determined by the data {g6 , h·, ·i, ∇, g0 } has nonnegative curvature. Additionally, the boundary of a small ball about the soul (zero-section) of (E, g E ) has positive intrinsic curvature, which proves one direction of Theorem B(1). Next, we describe some ways in which Theorem B overlaps known results related to connection metrics of nonnegative and positive curvature. • One direction of Theorem B(1), namely, that positive curvature implies the inequality, follows from the argument in [16] by which Strake and Walschap established inequality (1.2). We elaborate on this remark in Section 7. • Theorem B(2) is an improvement of Strake and Walschap’s sufficient condition for a connection metric of nonnegative curvature, which is equivalent to our condition with the right-hand side of the inequality multiplied by 1/2. • Theorem B(2) in the case where k = 2 and the vector bundle is oriented was done by Strake and Walschap in [16]. Theorem B(1) in this case follows from Strake and Walschap’s work and also appears explicitly in [4]. In this case, R ∇ can be identified with the 2-form  on 6 given by (X, Y ) = hR ∇ (X, Y )W, J W i, where |W | = 1 and J denotes the almost complex structure on E. The inequality of Theorem B becomes

80

KRISTOPHER TAPP

(D X (X, Y ))2 < |i X |2 · k6 (X, Y ), where i X  = (X, ·). Also, inequality (1.2) becomes (D X (X, Y ))2 ≤ |i X |2 · (k6 (X, Y ) − (3/4) 2 (X, Y )2 ); Yang proved in [20, Lem. 1] that this last inequality, together with the inequality k6 (X, Y ) ≥ (3/4) 2 (X, Y )2 , provides a necessary and sufficient condition for the connection metric g on E 1 with fibers of length 2π to have nonnegative curvature. Since, when k = 2, the sphere bundle E 1 is a principal bundle, nonnegative curvature on the sphere bundle implies nonnegative curvature on the vector bundle. • The strict inequality of Theorem B implies that R ∇ (W, V )X = 0 only when X = 0 or W ∧ V = 0. This is equivalent to saying that the induced connection in the sphere bundle E 1 is “fat.” The concept of fatness was introduced by A. Weinstein in [19]. Among other restrictions, it implies that dim(6) is even and is greater than or equal to k, with equality possible only if dim(6) is 2, 4, or 8. A. Derdzi´nski and A. Rigas proved in [6] that the only S 3 -bundle over S 4 which admits a fat connection is the Hopf bundle S 3 → S 7 → S 4 . This result rules out the possibility of using Theorem B to produce metrics of positive curvature on 7-dimensional exotic spheres. We refer the reader to [21] for a survey of results related to fatness. Because of the fatness implication, the strict inequality of Theorem B should probably be considered much stronger than the nonstrict inequality of equation (1.1). We return to the general problem of finding sufficient conditions for nonnegative curvature on E and for positive curvature on E 1 . The inequality of Theorem A is a relationship between the different curvatures that are visible at the soul, namely, the curvatures of 2-planes tangent to 6 (described by k6 ), the curvature R ∇ of the connection in ν(6), and the curvatures of “vertical” 2-planes (described by k F ). It is useful to write k F (W, V ) = hR F (W, V )V, W i = R F (W, V, V, W ), where R F , which we call the vertical curvature tensor, is just the restriction of the curvature tensor R of M to vectors in ν(6), so that (R F ) p : (ν p (6))4 → R. π More generally, a tensor R F on a vector bundle Rk → E → 6 such that, for each p ∈ 6, the map (R F ) p : (E p )4 → R has the symmetries of a curvature tensor (not necessarily including the Bianchi identity) is called a vertical curvature tensor on the bundle. We think of a vertical curvature tensor as prescribing the curvatures of vertical 2-planes at the zero-section. For p ∈ 6 and W, V ∈ E p , we write k F (W, V ) = R F (W, V, V, W ). As before, by parallel transporting W and V along geodesics from p in 6, we can think of k F (W, V ) as a real-valued function on a neighborhood of p in 6, and we write hessk F (W,V ) (X ) for the hessian of this function in the direction X ∈ T p 6. We prove that a strengthening of the necessary condition in Theorem A is sufficient to guarantee a metric of positive curvature on the sphere bundle.

CONDITIONS FOR NONNEGATIVE CURVATURE

THEOREM C

81

π

Let 6 be a compact manifold, and let Rk → E → 6 be a vector bundle over 6. If this bundle admits a metric g6 on 6, a Euclidean structure h·, ·i, a compatible connection ∇, and a vertical curvature tensor R F such that, for all p ∈ 6, X, Y ∈ T p 6 with X ∧ Y 6= 0, and W, V ∈ E p with W ∧ V 6= 0,
h(D X R ∇ )(X, Y )W, V i2 < |R ∇ (W, V )X |2 + (2/3) hessk F (W,V ) (X ) · k6 (X, Y ), then the unit-sphere bundle E 1 of E admits a metric of positive curvature. Some comments about Theorem C are in order. • To prove Theorem C, we construct a metric g E on E for which the boundary of a small ball about the zero-section has positive curvature. We believe that g E can always be constructed to be a complete metric of nonnegative curvature, but we are able to prove this only in the special case of connection metrics, as described in Theorem B. • L. Guijarro and Walschap proved that if a vector bundle admits a metric of nonnegative curvature, then so does the associated sphere bundle (see [10]). This is because the boundary of a small ball about the soul is convex and, hence, nonnegatively curved in the induced metric. Our theorems address the question of when this induced metric on the sphere bundle has positive curvature. For a metric of nonnegative curvature on a vector bundle, the inequality in Theorem A must hold; if, in addition, this inequality is strict on orthonormal vectors {X, Y, W, V }, then the induced metric on the sphere bundle must have positive curvature. • The strict inequality implies that |R ∇ (W, V )X |2 ≥ −(2/3) hessk F (W,V ) (X ) with equality only when X = 0 or W ∧ V = 0. This can be thought of as a generalized fatness condition. Because of the added generality, Derdzi´nski and Rigas’s result does not rule out the possibility of using Theorem C to find metrics of positive curvature on 7-dimensional exotic spheres. • In Theorem A, if the metric on M is such that each fiber of the projection π : M → 6 is radially symmetric (although not necessarily totally geodesic), then (2/3)k F (W, V ) = f ( p)·|W ∧V |2 for some function f : 6 → R. In other words, all vertical 2-planes at a fixed point p ∈ 6 have the same sectional curvature, so the vertical curvature information can be described entirely by a function f on 6. In this case, the inequality of Theorem A becomes
h(D X R ∇ )(X, Y )W, V i2 ≤ |R ∇ (W, V )X |2 +|W ∧V |2 ·hess f (X ) ·k6 (X, Y ). (1.3) π Conversely, if a vector bundle Rk → E → 6 admits structure {g6 , h·, ·i, ∇, f } (where {g6 , h·, ·i, ∇} are as in Theorem C and where f :

82

KRISTOPHER TAPP

6 → R) such that inequality (1.3) is satisfied and is strict for orthonormal vectors {X, Y, W, V }, then Theorem C provides a metric of positive curvature on E 1 for which the fibers are round (with varying diameters). • Both the inequality of Theorem A and its sharpening to a strict inequality for orthonormal vectors have natural interpretations. As we show, a mixed 2-plane σ at the soul is a critical point of the sectional curvature function, sec, on the Grassmannian G of 2-planes on M. The inequality in Theorem A comes from the fact that, since M has nonnegative curvature, the hessian of sec at σ must be nonnegative definite. The strictness of the inequality means that the only vectors in Tσ G contained in the nullspace of the hessian of sec are the vectors forced to be there by Perelman’s theorem. Our paper is organized as follows. In Sections 2 and 3 we describe the derivatives at the soul of the A and T tensors of the Riemannian submersion from an open manifold of nonnegative curvature onto its soul. This allows us in Section 4 to describe the hessian of sec at a mixed 2-plane σ at the soul. Theorem A is a consequence of this discussion. In Section 5 we describe how to construct a metric on a vector bundle from the data {g6 , h·, ·i, ∇, R F } prescribed in Theorem C. Our construction yields a warped connection metric, which means a metric formed from a connection metric by altering the fiber metrics, so that in our case the curvatures of the fibers at the zero-section are as prescribed by R F . In Section 6 we prove Theorem C by constructing a warped connection metric on a vector bundle so that the intrinsic metric on the boundary of a small ball about the zero-section has positive curvature. Unfortunately, we do not know how to verify that the warped connection metric itself has nonnegative curvature. But in Section 7 we at least show how to do this in the case of connection metrics and, thus, prove Theorem B. 2. Background: The metric near the soul In this section, M denotes an open manifold with nonnegative curvature, and 6 ⊂ M denotes a soul of M in the sense of [5]. Let ∇ denote the connection in the normal bundle ν(6) of 6 in M, and let R ∇ denote its curvature tensor. Let R denote the curvature tensor of M, and denote (E 1 , E 2 , E 3 , E 4 ) = R(E 1 , E 2 , E 3 , E 4 ) = hR(E 1 , E 2 )E 3 , E 4 i and k(E 1 , E 2 ) = (E 1 , E 2 , E 2 , E 1 ). Define R6 and R F (resp., k6 and k F ) as the restrictions of R (resp., k) to T 6 and ν(6). Our proof of Theorem A relies heavily on Perelman’s resolution of the soul conjecture in [14], which states the following.

CONDITIONS FOR NONNEGATIVE CURVATURE

83

2.1 (Perelman) The metric projection π : M → 6, which sends each point p ∈ M to the point π( p) ∈ 6 to which it is closest, is a well-defined Riemannian submersion. For any p ∈ 6, X ∈ T p 6, and V ∈ ν p (6), the surface (s, t) 7→ exp(s · V (t)) (s, t ∈ R, s ≥ 0), where V (t) denotes the parallel transport of V along the geodesic with initial tangent vector X , is a flat and totally geodesic half-plane. (We refer to these surfaces as Perelman flats.)

THEOREM

(1) (2)

One consequence of Perelman’s theorem is that mixed 2-planes at the soul are flat. 2.2 Let p ∈ 6, X, Y ∈ T p 6, and U ∈ ν p (6). Then (1) R(X, U )U = R(U, X )X = 0; in particular, k(X, U ) = 0; (2) R(X, Y )U = 2R(X, U )Y . COROLLARY

Proof Part (1) was originally proved by Cheeger and Gromoll [5, Th. 3.1]. Today it is obvious from Perelman’s theorem. (At least it is obvious that k(X, U ) = 0, but on a manifold of nonnegative curvature this implies that R(X, U )U = R(U, X )X .) Part (2) is found in [15]. It is a consequence of the Bianchi identity, R(X, Y )U = R(X, U )Y − R(Y, U )X, together with the vanishing of the mixed curvatures, which means that 0 = R(X + Y, U )(X + Y ) = R(X, U )Y + R(Y, U )X + 0 + 0. Although π has only been proved to be C 2 (see [9]), it is clearly C ∞ in a neighborhood of 6. We denote by A and T the fundamental tensors associated to π, as defined, for example, in [3, Chap. 9]. We collect in the following lemma some facts about the A and T tensors at the soul. All but part (4) of this lemma are well known. LEMMA 2.3 Let p ∈ 6, X, Y ∈ T p 6, and U, V, W ∈ ν p (6); (1) A X Y = A X U = 0; (2) TU X = TU V = 0; (3) (DV A) X Y = −(1/2)R ∇ (X, Y )V and (DV A) X U = (1/2)R ∇ (V, U )X ; (4) (DW T )U V = (DW T )U X = 0.

Proof Part (1) is obvious. For part (2), TU X = 0 because the Perelman flat through X and

84

KRISTOPHER TAPP

U is totally geodesic. Since hTU V, X i = −hTU X, V i = 0, it follows that TU V = 0 as well. Part (3) is an immediate consequence of [15, Prop. 1.7]. For part (4), first notice that (DW T )W X = 0 because the Perelman flat through X and W is totally geodesic. Next, since 0 = h(DW T )W X, U i = −h(DW T )W U, X i = −h(DW T )U W, X i, it follows that (DW T )U W = 0 as well. This is a special case of part (4). To get the general case from this special case, we apply O’Neill’s formula (see [3, Th. 9.28(b)]) and Corollary 2.2 as follows: 0 = (X, W, W, U ) = −h(DW T )U W, X i + h(DU T )W W, X i = h(DU T )W W, X i. It follows from this that (DU T )W W = 0. Since (DU T )W1 W2 is symmetric in W1 and W2 , it follows that DU T = 0. A different, more illuminating proof of Lemma 2.3(4) appears later in our proof of Lemma 5.2(3). The following formula for the curvature of an arbitrary 2-plane at the soul appears in [18, p. 615]. PROPOSITION 2.4 (Walschap) Let p ∈ 6, X, Y ∈ T p 6, and U, V ∈ ν p (6). Then

k(X + U, Y + V ) = k6 (X, Y ) + k F (U, V ) − 3(X, Y, U, V ). Proposition 2.4 is proved by expanding the left-hand side by linearity and by noticing that many of the resulting terms vanish by Corollary 2.2 and the fact that the soul is totally geodesic. COROLLARY 2.5 (Walschap) For all p ∈ 6, X, Y ∈ T p 6, and U, V ∈ ν p (6),

9(X, Y, U, V )2 ≤ 4k6 (X, Y ) · k F (U, V ). Proof Fix X, Y ∈ T p 6 and U, V ∈ ν p (6). By Proposition 2.4, k6 (X, Y ) + k F (U, V ) − 3(X, Y, U, V ) ≥ 0. Of course, this inequality must remain true for any rescalings x X, yY, uU, vV of the vectors (x, y, u, v ∈ R). In other words, Q(x y, uv) ≥ 0, where Q is the quadratic

CONDITIONS FOR NONNEGATIVE CURVATURE

85

form with matrix  k6 (X, Y ) −(3/2)(X, Y, U, V ) Q= . −(3/2)(X, Y, U, V ) k F (U, V ) 

Hence, Q is nonnegative definite and, therefore, has nonnegative determinate. 3. The second derivative of the T -tensor at the soul The only new observation in Section 2 is that the first derivative, DT , of the T -tensor of π vanishes at points of 6. The main goal of this section is to describe the second derivative, D 2 T . Since the T -tensor measures the failure of the fibers to be mutually isometric, one might expect that D 2 T at 6 measures the failure of the fibers to look the same at the soul, in other words, the failure of R F (W1 , W2 , W3 , W4 ) to be constant on a path in 6 along which the sections Wi of ν(6) are parallel. This intuition is essentially right, although it is cleaner to describe D 2 T in terms of the symmetrization, ˚ F , of R F . R Therefore, we begin with a discussion of symmetrization. If V is a vector space with orthonormal basis {e1 , . . . , ek } and if R is a curvature tensor on V, then R˚ : S 2 V → S 2 V commonly denotes the induced tensor on symmetric 2-forms, namely, X ˚ R(h)(U, V) = R(ei , U, V, e j ) · h(ei , e j ). 1≤i, j≤k

˚ 1 , W2 , U, V ) = R(h)(U, ˚ It is also useful to define R(W V ), where h ∈ S 2 V is h(e, f ) =

1 (he, W1 ih f, W2 i + he, W2 ih f, W1 i). 2

In this way, we consider R˚ to be a tensor of order 4 on V, which has the following simple description:
˚ 1 , W2 , U, V ) = 1 R(W1 , U, V, W2 ) + R(W2 , U, V, W1 ) . R(W 2 ˚ follow from the symmetries of R: The following symmetries of R ˚ 1 , W2 , U, V ) = R(W ˚ 2 , W1 , U, V ) = R(W ˚ 1 , W2 , V, U ) = R(U, ˚ R(W V, W1 , W2 ), (3.1) ˚ R(W, W, W, U ) = 0. ˚ when V is the tangent space, T p M, of a We now give a useful description of R Riemannian manifold, and when R is the curvature tensor given by the Riemannian metric on M. First, for W, U, V ∈ T p M, define F(W, U, V ) = hd expW U, d expW V i.

(3.2)

86

KRISTOPHER TAPP

LEMMA 3.1 ˚ can be described in terms of F as follows: R

  2 ˚ 1 , W2 , U, V ) = − 3 d R(W F(t W1 + sW2 , U, V ). 2 ds dt s=t=0 Proof Let ϒ(W1 , W2 , U, V ) denote the right-hand side of the equation, which we wish to ˚ 1 , W2 , U, V ). Since both ϒ and R ˚ are symmetric in W1 , W2 and also prove equals R(W ˚ in U, V , it suffices to prove that R(W, W, U, U ) = ϒ(W, W, U, U ) for all W, U ∈ T p M. It is straightforward to see that ˚ R(W, W, U, U ) = R(W, U, U, W ) = k(W, U ). Therefore, it remains to prove that 2 ϒ(W, W, U, U ) = − k(W, U ) for all vectors W, U . 3 In fact, since ϒ(W, W, W, U ) = 0, it suffices to verify this when W and U are orthonormal, in which case k(W, U ) is the sectional curvature of the 2-plane σ which they span. Let S denote the surface in M obtained as the exponential image of σ . Write the metric on S in polar coordinates: ds2 = dr2 + f 2 (r, θ) dθ 2 , where θ = 0 corresponds to the direction of W . Let γ (r ) = exp(r W ). (In polar coordinates, γ (r ) = (r, 0).) Along γ , f can be expressed as p f (r, 0) = r F(r W, U, U ). The Gauss curvature of S at (r, 0) equals − frr (r, 0)/ f (r, 0), where frr denotes the second partial with respect to r . The result now follows by performing the differentiation and taking the limit as r → 0. We return to our setup where M is an open manifold with nonnegative curvature, 6 ⊂ M is a soul, and R F is the vertical curvature tensor. Let R˚ F denote the fiberwise symmetrization of R F as described above. When the vectors Wi ∈ ν p (6) are ˚ F (W1 , W2 , W3 , W4 ) as a real-valued function on 6 near p fixed, we can think of R (by parallel transporting the Wi along geodesics in 6 from p). The following lemma describes D 2 T in terms of the gradient of this real-valued function.

CONDITIONS FOR NONNEGATIVE CURVATURE

87

LEMMA 3.2 For all p ∈ 6 and W1 , W2 , U, V ∈ ν p (6),

(DW1 DW2 T )U V =

1 ˚ 1 , W2 , U, V ). grad R(W 3

Proof We first establish a convention for lifting vectors. If X ∈ T p 6, we denote by X¯ an extension of X to a basic vector field on M. Additionally, for U ∈ ν p (6), let U¯ be the extension U to a vertical vector field on M in a neighborhood of p constructed as follows. First, extend U to a vector field along the fiber π −1 ( p) in a neighborhood of p by defining, for each W ∈ ν p (6) with small norm, U¯ |exp(W ) = d expW (U ). Then extend U to a section of ν(6) near p by parallel transporting U along geodesics in 6 from p. Finally, for each point q ∈ 6 near p, we extend the vector field along the fiber π −1 (q) in the same way we extended it to π −1 ( p). We begin by proving that, for any X ∈ T p 6 and U, V, W ∈ ν p (6) with |W | small, 1 hTV¯ X¯ , U¯ i = X F(W, U, V ), (3.3) exp(W ) 2 where F (which is defined in equation (3.2)) is thought of as a real-valued function on 6 near p by parallel transporting W, U, V along geodesics in 6 from p. Notice that X¯ and U¯ commute simply because their preimages under d exp⊥ in T (ν(6)) commute. (Here exp⊥ : ν(6) → M denotes the normal exponential map.) So, letting p¯ = exp(W ) and using the standard coordinate-free expression for the connection, we see that 2hTV¯ X¯ , U¯ i p¯ = 2h∇V¯ X¯ , U¯ i p¯ = X¯ hV¯ , U¯ i p¯ + V¯ hU¯ , X¯ i p¯ − U¯ h X¯ , V¯ i p¯ − h[ X¯ , U¯ ], V¯ i p¯ − h[U¯ , V¯ ], X¯ i p¯ − h[ X¯ , V¯ ], U¯ i p¯ = X¯ hV¯ , U¯ i p¯ = X F(W, U, V ). This verifies equation (3.3). It follows easily that, for any W, U, V ∈ ν p (6) with |W | small, 1 (TV¯ U¯ ) = − grad F(W, U, V ). (3.4) exp(W ) 2 Finally, we use equation (3.4) to study D 2 T . To prove the lemma, it suffices to 2 T ) V = (1/3) grad R(W, ˚ verify that (DW W, U, V ) for all U, V, W ∈ ν p (6), which U

88

KRISTOPHER TAPP

is done as follows. Let γ (t) = exp(t W ). Then D (D ¯ T ¯ V¯ )γ (t) − 0 − 0 dt t=0 W U D  D = (T ¯ V¯ )γ (t+r ) − (T(∇W¯ U¯ ) V¯ )γ (t) dt t=0 dr r =0 U
 − T ¯ (∇ ¯ V¯ )

DW (DW TU V ) =

U

W

γ (t)

D2

(TU¯ V¯ )γ (t) − 0 − 0 dt2 t=0
1 D2 =− grad F(t W, U, V ) γ (t) 2 2 dt t=0 1 D2 =− grad F(t W, U, V ) 2 dt2 t=0  d2  1 = − grad 2 F(t W, U, V ) 2 dt t=0 1 ˚ = grad R(W, W, U, V ). 3 =

To justify the third equality above, notice that D (T ¯ V¯ )γ (t) = (DW T )(∇W U¯ ) V + T(∇W ∇ ¯ U¯ ) V + T(∇W U¯ ) (∇W V¯ ) W dt t=0 (∇W¯ U ) = 0 + 0 + 0 = 0. 4. A necessary condition for nonnegative curvature on vector bundles In this section we prove Theorem A by studying the derivatives of a function that records the curvatures of a family of 2-planes. The family begins with a mixed 2plane at the soul and then drifts so that the base point moves away from the soul while the 2-plane simultaneously twists away from being a mixed 2-plane. More precisely, the setup for this section is as follows. Let p ∈ 6, X, Y ∈ T p 6, and W, V, U ∈ ν p (6). Let γ (t) := exp(t W ), and let X t , Yt , Ut , Vt denote the parallel transports of X, Y, U , and V along γ (t). By Perelman’s theorem, X t and Yt are horizontal for all t ∈ [0, ∞). In other words, parallel translation along the radial geodesic γ preserves the horizontal space. Therefore, it must also preserve the vertical space, so Ut and Vt are vertical for all t ∈ [0, ∞). Define 9(t) = 9 X Y U V W (t) = k(X t + tUt , tYt + Vt ),

(4.1)

which is the unnormalized sectional curvature of the 2-plane based at γ (t) spanned by X t + tUt and tYt + Vt . The special case of this construction when U = Y = 0 was studied by V. Marenich in [12]. Notice that 9(0) = k(X, V ) = 0 by Corollary 2.2.

CONDITIONS FOR NONNEGATIVE CURVATURE

89

The goal of this section is to derive formulas for 9 0 (0) and 9 00 (0). Toward this end, we write  9(t) = (X t , Vt , Vt , X t ) + t · 2(X t , Yt , Vt , X t ) + 2(X t , Vt , Vt , Ut )  + t 2 · (X t , Yt , Yt , X t ) + (Ut , Vt , Vt , Ut ) + 2(X t , Yt , Vt , Ut ) + 2(X t , Vt , Yt , Ut )  + t 3 · 2(Ut , Yt , Vt , Ut ) + 2(X t , Yt , Yt , Ut ) + t 4 · (Ut , Yt , Yt , Ut ). (4.2) PROPOSITION 4.1 We have 9 0 (0) = 0.

Proof Since M has nonnegative curvature, 9 0 (0) ≥ 0. But if it were the case that 9 0 (0) > 0, then replacing W with −W would yield 9 0 (0) < 0. Hence, 9 0 (0) = 0. In order that our proof generalize properly in Section 5, we also compute 9 0 (0) directly. From equation (4.2), 9 0 (0) = 2(X, Y, V, X ) + 2(X, V, V, U ) +

d (X t , Vt , Vt , X t ), dt t=0

but (X, Y, V, X ) = 0 because the soul is totally geodesic, and (X, V, V, U ) = 0 by Corollary 2.2. We use O’Neill’s formula (see [3, Th. 9.28(c)]) to study the third term ([3] uses a different curvature sign convention): d (X t , Vt , Vt , X t ) dt t=0 d  h(D X t T )Vt Vt , X t i − hTVt X t , TVt X t i + hA X t Vt , A X t Vt i = dt t=0 d = h(D X t T )Vt Vt , X t i − 0 + 0 dt t=0 DD
E = (D X t T )Vt Vt , X = h(DW D X T )V V, X i = h(D X DW T )V V, X i = 0. dt PROPOSITION

4.2

We have 1 9 00 (0) = 2k6 (X, Y ) + 2k F (U, V ) − 6hR ∇ (X, Y )U, V i + |R ∇ (W, V )X |2 2 1 − 2h(D X R ∇ )(X, Y )W, V i + hessk F (W,V ) (X ) 3 4 4 ˚ ˚ + D X R(W, U, V, V ) − D X R(W, V, U, V ). 3 3

90

KRISTOPHER TAPP

Proof From equation (4.2),  9 00 (0) = 2 (X, Y, Y, X ) + (U, V, V, U ) + 2(X, Y, V, U ) + 2(X, V, Y, U ) d  +2 2(X t , Yt , Vt , X t ) + 2(X t , Vt , Vt , Ut ) dt t=0 d2 + 2 (X t , Vt , Vt , X t ). dt t=0 The top line of this expression can be simplified using Corollary 2.2(2): (X, Y, Y, X ) + (U, V, V, U ) + 2(X, Y, V, U ) + 2(X, V, Y, U ) = k6 (X, Y ) + k F (U, V ) − 3(X, Y, U, V ). Next, from one of O’Neill’s formulas (see [3, Th. 9.28(e)]), d (X t , Yt , Vt , X t ) dt t=0 d  = h(D X t A) X t Yt , Vt i + 2hA X t Yt , TVt X t i dt t=0 DD
E d (D X t A) X t Yt , V = h(D X t A) X t Yt , Vt i = dt t=0 dt = h(DW D X A) X Y, V i = h(D X DW A) X Y, V i 1 = − h(D X R ∇ )(X, Y )W, V i. 2 We apply another of O’Neill’s formulas (see [3, Th. 9.28(b)]) to simplify the next term: d (X t , Vt , Vt , Ut ) dt t=0 d  h(DVt T )Ut Vt , X t i − h(DUt T )Vt Vt , X t i =− dt t=0 = −h(DW DV T )U V, X i + h(DW DU T )V V, X i 1 1 ˚ ˚ V, U, V ), X i + hgrad R(W, U, V, V ), X i = − hgrad R(W, 3 3 1 1 ˚ ˚ = − D X R(W, V, U, V ) + D X R(W, U, V, V ). 3 3

CONDITIONS FOR NONNEGATIVE CURVATURE

91

Finally, d2 (X t , Vt , Vt , X t ) dt2 t=0 d2  h(D X t T )Vt Vt , X t i − hTVt X t , TVt X t i + hA X t Vt , A X t Vt i = 2 dt t=0 2 = h(DW D X T )V V, X i − 2h(DW T )V X, (DW T )V X i + 2h(DW A) X V, (DW A) X V i 1 2 = h(D X DW T )V V, X i − 0 + |R ∇ (W, V )X |2 . 2 Theorem A is an immediate corollary of Proposition 4.2, as we now show. Proof of Theorem A Let p ∈ 6, X, Y ∈ T p 6, and U, V, W ∈ ν p (6). Since M has nonnegative curvature, 9 X00 Y U V W (t) ≥ 0. In particular, this is true when U = 0, which implies that the following expression is nonnegative: 1 1 2k6 (X, Y ) + |R ∇ (W, V )X |2 − 2h(D X R ∇ )(X, Y )W, V i + hessk F (W,V ) (X ). 2 3 Of course, the same remains true for any rescalings x X, yY, wW, vV of the vectors (x, y, w, v ∈ R). In other words, Q(x y, xwv) ≥ 0, where Q is the quadratic form with matrix   2k6 (X, Y ) h(D X R ∇ )(X, Y )W, V i Q= . h(D X R ∇ )(X, Y )W, V i (1/2)|R ∇ (W, V )X |2 + (1/3) hessk F (W,V ) (X ) Hence, Q is nonnegative definite, and its determinate is, therefore, nonnegative. This implies that   2 h(D X R ∇ )(X, Y )W, V i2 ≤ |R ∇ (W, V )X |2 + hessk F (U,V ) (X ) · k6 (X, Y ). 3 We end by mentioning a more general possible definition of 9, namely, 9(t) = 9 X Xˆ Y U V Vˆ W (t) = k(X t + t Xˆ t + tUt , tYt + Vt + t Vˆt ).

(4.3)

Although this seems more general, it is easy to see that 9 0 ˆ (0) = 0 X X Y U V Vˆ W 00 00 and 9 ˆ (0) = 9 X Y U V W (0). In other words, our derivative formulas do not X X Y U V Vˆ W notice the added generality.

92

KRISTOPHER TAPP

5. Warped connection metrics In this section we define and study a class of metrics on vector bundles called warped connection metrics, which are more general than connection metrics. Given a connection metric g E on the total space E of a vector bundle, Rk → π E → 6, we write TE =H ⊕V ⊕r for the orthogonal decomposition of the tangent bundle of E, where H is the distribution determined by ∇, r is the span of gradient of the distance to the zero-section (r is 1-dimensional on E − 6 and k-dimensional on 6), and V describes the space of vectors tangent to the fibers of π and orthogonal to r. We make the following definition. Definition 5.1 π A warped connection metric g E on a vector bundle Rk → E → 6 is any smooth metric obtained by starting with a connection metric and then altering the metric arbitrarily on V . For a warped connection metric, it is easy to see that π is still a Riemannian submersion and that exp : ν(6) → E is still a diffeomorphism. Also, the zero-section, 6, is totally geodesic, and both statements of Perelman’s theorem (see Th. 2.1) are valid. We consider the following structures on (E, g E ), all defined analogously to the way ˚ F, they were defined for nonnegatively curved metrics: g6 , k6 , ∇, R ∇ , F, R F , k F , R A, T , and 9. For example, F (which we call the warping function) is defined by the equation F p (W, U, V ) = hd expW U, d expW V i for p ∈ 6 and W, U, V ∈ E p .

(5.1)

Notice that the vectors d expW U and d expW V are both tangent to the fibers of π; hence, F records the metrics of the fibers. F is a smooth function from {( p, W, U, V ) | p ∈ 6 and W, U, V ∈ ν p (6)} to R, and F has the following properties: (1) F(W, ·, ·) is a symmetric positive-definite bilinear form for each W ; (2) F(W, W, U ) = F(0, W, U ) = hW, U i; d (3) dt t=0 F(t W, U, V ) = 0; 2 d2 (4) F(t W1 + sW2 , U, V ) = d F(tU + sV, W1 , W2 ). ds dt s=t=0

ds dt s=t=0

Perelman’s theorem implies that any complete metric of nonnegative curvature on a vector bundle agrees with a warped connection metric inside of the cut-locus of the soul. (This is because if the normal bundle, ν(6), of the soul 6 in M is endowed with its natural connection metric, then exp : ν(6) → M preserves horizontal and vertical

CONDITIONS FOR NONNEGATIVE CURVATURE

93

spaces and is an isometry on H and r.) Guijarro proved in [8] that a metric of nonnegative curvature on a vector bundle can always be altered so that exp : ν(6) → M becomes a diffeomorphism; this altered metric is a warped connection metric. So, the class of warped connection metrics is general enough to resolve Cheeger and Gromoll’s question; that is, if a vector bundle admits a metric of nonnegative curvature, then it admits a warped connection metric of nonnegative curvature. On the other hand, the class of warped connection metrics is fairly rigid. The next lemma says that warped connection metrics share much of the important structure of nonnegatively curved metrics. 5.2 For a warped connection metric g E on the total space E of a vector bundle π Rk → E → 6, the following are true for all p ∈ 6, X, Xˆ , Y ∈ T p 6, and W, W1 , W2 , U, V, Vˆ ∈ E p : (1) A p = 0 and T p = 0; (2) (DV A) X Y = −(1/2)R ∇ (X, Y )V and (DV A) X U = (1/2)R ∇ (V, U )X ; ˚ 1 , W2 , U, V ); (3) DT p = 0 and (DW1 DW2 T )U V = (1/3) grad R(W LEMMA

(4) (5) (6) (7) (8)

R(X, V )V = R(V, X )X = 0 and R(X, Y )U = 2R(X, U )Y ; k(X + U, Y + V ) = k6 (X, Y ) + k F (U, V ) − 3(X, Y, U, V ); 9 X0 Y U V W (0) = 9 0 ˆ (0) = 0; X X Y U V Vˆ W 00 00 9 X Y U V W (0) = 9 ˆ (0) is given by the equation of Proposition 4.2; X X Y U V Vˆ W the boundary of a sufficiently small ball about 6 is convex.

Proof This lemma essentially follows from previous arguments, but one alteration is needed. For nonnegatively curved metrics, the fact that k(X, V ) = 0 implies that R(X, V )V = R(V, X )X = 0, which in turn was used to prove that DT p = 0. For general warped connection metrics, k(X, V ) = 0, but this does not automatically imply that R(X, V )V = R(V, X )X = 0. We must prove things in a different order. First, we show that DT p = 0. Using O’Neill’s formula, this then implies that R(X, V )V = R(V, X )X = 0. To show that DT p = 0, let γ (t) = exp⊥ (t W ). Then
1 D D grad F(t W, U, V ) γ (t) (TU¯ V¯ )γ (t) = − dt t=0 2 dt t=0 d  1 D 1 =− grad F(t W, U, V ) = − grad F(t W, U, V ) = 0. 2 dt t=0 2 dt t=0

DW TU V =

Part (8) was proved for nonnegatively curved metrics by Guijarro and Walschap in [10], and their proof remains valid for general warped connection metrics.

94

KRISTOPHER TAPP

Just as a connection metric on a vector bundle is prescribed by a Euclidean structure, a connection, a metric on the base space, and a rotationally symmetric metric on Rk , a warped connection metric can also be prescribed by structures on the bundle. Suppose π that Rk → E → 6 is a vector bundle. Let g6 be a metric on 6. Let F be any smooth function from {( p, W, U, V ) | p ∈ 6 and W, U, V ∈ ν p (6)} to R which has the following two properties: (1) F p (W, ·, ·) is a symmetric positive-definite bilinear form for each p ∈ 6 and each W ∈ E p ; (2) F p (W, W, U ) = F p (0, W, U ) for each p ∈ 6 and each W, U ∈ E p . We call F a warping function. The following properties follow from the above two: d (3) dt t=0 F(t W, U, V ) = 0; 2 d2 (4) F(t W1 + sW2 , U, V ) = d F(tU + sV, W1 , W2 ). ds dt s=t=0

ds dt s=t=0

To see this, notice that, by property (1), F induces a smooth metric on each fiber E p as follows: hU, V i = F p (W, U, V ), where U, V ∈ TW E p , and TW E p is identified with E p in the obvious manner. By property (2), the identity map from E p to E p is the exponential map with respect to this metric. Properties (3) and (4) are now familiar facts about metrics in polar coordinates. F determines a Euclidean structure on the bundle as follows: h·, ·i = F(0, ·, ·). Suppose that ∇ is a connection compatible with this Euclidean structure. Then there exists a unique warped connection metric g E on E for which π : (E, g E ) → (6, g6 ) is a Riemannian submersion with horizontal distribution determined by ∇ and with fiber metrics determined by F as described above, so that F(W, U, V ) = hd expW U, d expW V i for all p ∈ 6 and all W, U, V ∈ E p . To construct g E , begin with the connection metric with flat fibers determined by g6 , h·, ·i, and ∇, and then alter the fiber metrics according to F. We call g E the warped connection metric on E determined by the data {g6 , ∇, F}. The warping function F can itself be prescribed in terms of more basic structures. More precisely, suppose that h·, ·i is a Euclidean structure on a vector bundle, and ˚ F be the suppose that R F is a vertical curvature tensor on the vector bundle. Let R fiberwise symmetrization of R F , as described in Section 3. Define 1 F(W, U, V ) = hU, V i − R˚ F (W, W, U, V ). 3 It is easy to verify that F is smooth and has properties (1) and (2) above. We call the resulting metric g E the warped connection metric on E determined by the data {g6 , h·, ·i, ∇, R F }, even though g E is a nondegenerate metric only in a neighborhood of the zero-section and not necessarily on all of E. ˚ 0 ) denote the vertical curvature tensor (and its symmetrization) If R 0F (and R F associated with this warped connection metric, it is clear from Lemma 3.1 that ˚0 = R ˚ F . It follows that R F (W, V, V, W ) = R 0 (W, V, V, W ) for all p ∈ 6 and all R F F

CONDITIONS FOR NONNEGATIVE CURVATURE

95

W, V ∈ E p . If R F satisfies the Bianchi identity (which we need not assume for the previous discussion), then this implies that R F = R 0F . In other words, we succeeded in prescribing the fiber metric so that at the zero-section its curvature tensor is R F . 6. Proof of Theorem C π Suppose that Rk → E → 6 is a vector bundle over 6 which admits the structures {g6 , h·, ·i, ∇, R F } so that the inequality of Theorem C is satisfied. We wish to choose a warped connection metric g E on E such that a small sphere about 6 has positive curvature. An obvious first try is the warped connection metric determined by the data {g6 , h·, ·i, ∇, R F }. However, this turns out not to work. The problem is that, since only the hessian of R F appears in the inequality of Theorem C, this choice provides no control over how large the vertical sectional curvatures at the zero-section are. For example, in the connection metric case, the inequality is satisfied for R F = 0, but using flat fibers is clearly a poor choice. Therefore, we modify R F to boost the sectional curvatures of vertical 2-planes at the zero-section. When C is a real number, let RC denote the vertical curvature tensor on the vector bundle which satisfies RC (W, U, U, W ) = C · |W ∧ U |2

for all p ∈ 6, and W, U ∈ E p .

In other words, for each p ∈ 6, (RC ) p is the curvature tensor corresponding to a point with constant sectional curvature C. Let g E be the warped connection metric on E which is determined by the data {g6 , h·, ·i, ∇, R 0F }, where R 0F = RC + R F . ˚ F and R ˚ 0 the symmetrizations of R F and R 0 . We denote by k F and We denote by R F F 0 k F the unnormalized sectional curvatures of R F and R 0F . Notice that hessk 0F (W,V ) = hessk F (W,V ) . We prove that, for sufficiently large C, the boundary of a sufficiently small ball about the zero-section of (E, g E ) has positive curvature. CLAIM 1 C can be chosen sufficiently large so that the curvature of every 2-plane at every point of the zero-section 6 of (E, g E ) is positive, except for the mixed 2-planes, whose curvatures are zero.

Proof By Lemma 5.2(5) and the proof of Corollary 2.5, this claim follows from the fact that C can easily be chosen so that 9(X, Y, U, V )2 ≤ 4k6 (X, Y ) · k 0F (U, V ) with equality only when X ∧ Y = 0 or U ∧ V = 0.

96

KRISTOPHER TAPP

CLAIM 2 C can be chosen so that if p ∈ 6, X, Y ∈ T p 6, and U, V, W ∈ E p are vectors for which |X | = |V | = 1, hW, U i = 0, and hW, V i = 0, then

9 X00 Y U V W (0) > 0. Proof By the hypothesis of Theorem C and a compactness argument, there exist  > 0 depending only on {g6 , h·, ·i, ∇, R F } so that h(D X R ∇ )(X, Y )W, V i2   2 ≤ (1 − ) |R ∇ (W, V )X |2 + hessk F (W,V ) (X ) · k6 (X, Y ) (6.1) 3 for all orthonormal vectors {X, Y, W, V } with X, Y ∈ T p 6 and W, V ∈ E p . But since equation (6.1) is invariant under rescalings of the vectors, projection of Y perpendicular to X , and projection of V perpendicular to W , this inequality is, in fact, valid for all (not necessarily orthonormal) vectors {X, Y, W, V }. By the argument of our proof of Theorem A, equation (6.1) implies that the following is true for all X, Y ∈ T p 6 and W, V ∈ E p : 1 2(1 − )k6 (X, Y ) + |R ∇ (W, V )X |2 2 − 2h(D X R ∇ )(X, Y )W, V i +

1 hessk F (W,V ) (X ) ≥ 0. (6.2) 3

Now, Lemma 5.2(7) says that 9 00 (0) is given by the formula of Proposition 4.2, which we can rewrite as follows: 1 9 00 (0) = 2(1 − )k6 (X, Y ) + |R ∇ (W, V )X |2 0 2 1 − 2h(D X R ∇ )(X, Y )W, V i + hessk F (W,V ) (X ) 3 + 2k6 (X, Y ) + 2k F (U, V ) − 6hR ∇ (X, Y )U, V i 4 ˚ 0F (W, U, V, V ) − 4 D X R˚ 0F (W, V, U, V ). + DX R 3 3

(6.3)

Now let X, Y ∈ T p 6 and U, V, W ∈ E p be vectors for which |X | = |V | = 1, hW, V i = 0, and hW, U i = 0. Also, assume that |W | = 1. We must prove that 9 00 (0) = 9 X00 Y U V W (0) > 0. Equation (6.2) says that the top two lines of equation (6.3) are nonnegative. Further, the hypothesis of Theorem C, together with a compactness argument, gives 1 1 ∇ |R (W, V )X |2 + hessk F (W,V ) (X ) > δ > 0 2 3

CONDITIONS FOR NONNEGATIVE CURVATURE

97

for some δ > 0 depending only on {g6 , h·, ·i, ∇, R F }. It follows that there exist constants δ1 , δ2 > 0 such that if |Y | ≤ δ1 , then the sum of the terms on the first two lines of equation (6.3) is greater than δ2 . Let H denote the sum of the terms of the last two lines of equation (6.3). It suffices to choose C sufficiently large that (1) if |Y | ≤ δ1 , then H + δ2 ≥ 0, and (2) if |Y | > δ1 , then H > 0. ˚ F and that But notice that D X R˚ 0F = D X R ˚ 0F (U, U, V, V ) = R ˚ F (U, U, V, V ) + C|U ∧ V |2 . k F (U, V ) = R From this it is straightforward to choose C large enough that the above two conditions are met. This completes the proof of Claim 2 under the added hypothesis that |W | = 1. 00 00 But by equation (6.3), it is clear that 9 X,aY,aU,V,aW (0) = a 2 9 X,Y,U,V,W (0), which allows us to drop the assumption that |W | = 1. We prove now that if C is chosen as large as required for Claims 1 and 2, then the boundary of a sufficiently small ball about 6 in (E, g E ) has positive extrinsic curvature. By Lemma 5.2(8), it must then have positive intrinsic curvature as well. It is useful to consider the following manifold:   = ( p, X, Y, U, V, W ) p ∈ 6; X, Y ∈ T p 6; U, V, W ∈ E p . Define f :  → R as follows: f ( p, X, Y, U, V, W ) = k( X¯ + U¯ , Y¯ + V¯ ), where { X¯ , Y¯ , U¯ , V¯ } are the lifts of {X, Y, U, V } to TW E (via parallel translation along W ). Notice that f has value zero on the compact submanifold N = {( p, X, 0, 0, V, 0) ∈  | |X | = |V | = 1}. By Lemma 5.2(6), N is a critical submanifold of f . Next, consider the following subset of :   = ( p, X, Y, U, V, W ) ∈  |W | < , hW, U i = 0 and hW, V i = 0 . Notice that N ⊂  . Even though  is not a smooth submanifold of , we can still make the following definition. Let T denote the collection of all vectors tangent to  at points of N which are also tangent to  . In other words, for n ∈ N and J ∈ Tn , J ∈ T if and only if there exists a path γ in  with γ 0 (0) = J such that γ ([0, δ]) ⊂ T for some δ > 0. By Claim 2, the Hessian of f at each point of N is positive definite in the directions of T . That is, for all J ∈ T , hess f (J ) > 0. Since the collection of unit

98

KRISTOPHER TAPP

directions in T is compact, it follows that there is a neighborhood of N in  on which f is strictly positive (except on N itself, where f is zero). Therefore, in the Grassmannian of 2-planes on M, there is a neighborhood of each mixed 2-plane at each point of the soul in which the curvature of every 2-plane tangent to a distance sphere about the soul is positive. But, by Claim 1, every nonmixed 2-plane at the soul has positive curvature, and, therefore, it also has such a neighborhood. This proves that small distance spheres about 6 in (E, g E ) have strictly positive curvature, which completes the proof. 7. Proof of Theorem B In this section, we prove Theorem B. As we mentioned in the introduction, Strake and Walschap showed by explicit computation that, for a connection metric g E of π nonnegative curvature on the total space E of a vector bundle Rk → E → 6, inequality (1.2) from our introduction is satisfied. Therefore, the weaker inequality (1.1) is satisfied as well. Their argument uses only the nonnegativity of 2-planes of the form span{X + V, Y }. These 2-planes are tangent to distance spheres about 6 in E. Also, their argument provides strict inequality when these 2-planes have strictly positive curvature. By the Gauss equation, the intrinsic curvature of such 2-planes (in the induced metric on the distance sphere) equals its extrinsic curvature in (E, g E ). Consequently, if some distance sphere about 6 in E has strictly positive curvature in the induced metric, then the inequality of Theorem B must be satisfied. This proves one direction of Theorem B(1). The second direction of Theorem B(1), as well as Theorem B(2), are proved next. We begin with a lemma. 7.1 π If a vector bundle Rk → E → 6 admits a warped connection metric g E which is nonnegatively curved in a neighborhood of the zero-section 6, then it admits a complete metric g 0E which has nonnegative curvature everywhere. If g E is a connection metric, then g 0E can be chosen to be a connection metric as well. LEMMA

Proof It follows from Lemma 5.2(8) that the main construction of [8] can be used to modify the metric g E into a complete metric g 0E with everywhere nonnegative curvature. (Alternately, use the main construction of [11], on which [8] is based.) Proof of Theorem B π Let Rk → E → 6 be a vector bundle with the structures {g6 , h·, ·i, ∇} satisfying the inequality of Theorem B. Let g E denote a connection metric on E determined by this data, whose fiber metric is chosen so that the curvature of every vertical 2-plane at

CONDITIONS FOR NONNEGATIVE CURVATURE

99

every point of the zero-section 6 equals C (e.g., Strake and Walschap used the fiber metric dr 2 + G 2 (r ) dσ 2 , where G 2 (r ) = 3r 2 /(3 + Cr 2 )). We wish to prove that, for large enough C, a neighborhood of 6 in (E, g E ) must have nonnegative curvature. Together with the lemma, this will complete the proof. Let p ∈ 6, and let W ∈ E p be a vector with small norm. Let X, Y ∈ T p 6, and let U, V ∈ E p with |X | = |Y | = |U | = |V | = 1, hW, U i = 0, and hW, V i = 0. Let X¯ , Y¯ , U¯ , V¯ denote lifts of X, Y, U, V to TW E (via parallel transport along W ). Let ∂r ∈ TW E denote the radial vector, by which we mean the unit-length vector pointing directly away from 6. Let α, β, γ , δ, ζ, η be positive real numbers. We need to ensure that k(α X¯ + β∂r + γ V¯ , δ∂r + ζ Y¯ + ηU¯ ) ≥ 0. Much of the work of this calculation was done by Strake and Walschap in [16]. As they showed, k(α X¯ + β∂r + γ V¯ , δ∂r + ζ Y¯ + ηU¯ ) = k(α X¯ + γ V¯ , ζ Y¯ + ηU¯ ) − k(α X¯ , ζ Y¯ ) + Q 3 (αζ, γ δ, βη), where Q 3 is the quadratic form with matrix  3 ¯ ¯ 3 ¯ ¯ ¯ ¯  k( X¯ , Y¯ ) 2 ( X , Y , ∂r , V ) − 2 ( X , Y , ∂r , U )  3 ( X¯ , Y¯ , ∂r , V¯ ) k(∂r , V¯ ) (∂r , V¯ , ∂r , U¯ )  . 2 3 − 2 (X, Y, ∂r , U¯ ) (∂r , V, ∂r , U¯ ) k(∂r , U¯ ) It follows from Strake and Walschap’s work that, for any value of  > 0, the constant C can be chosen so that Q 3 is nonnegative definite. Therefore, it remains to prove that  can be chosen small enough to ensure that k(α X¯ + γ V¯ , ζ Y¯ + ηU¯ ) − k(α X¯ , ζ Y¯ ) ≥ 0. We modify the argument in Theorem C by which we proved that small distance spheres are positively curved to get this little bit more that is required. We can use the value of  from equation (6.2), which simplifies in the connection metric case to 1 2(1 − )k6 (X, Y ) + |R ∇ (W, V )X |2 − 2h(D X R ∇ )(X, Y )W, V i ≥ 0. 2 Modify the proof of Theorem C by defining f :  → R as follows: f ( p, X, Y, U, V, W ) = k( X¯ + U¯ , Y¯ + V¯ ) − k( X¯ , Y¯ ). It is still easy to see that N is a critical submanifold of f and that the hessian of f is positive definite in directions of T . The proves that, in the Grassmannian of 2-planes on M, there is a neighborhood of each mixed 2-plane at each point of 6 on which the sectional curvature function is nonnegative. Since nonmixed 2-planes at points of 6 also have such neighborhoods, this proves that a neighborhood of 6 in (E, g E ) has nonnegative curvature, which completes the proof.

100

KRISTOPHER TAPP

Acknowledgments. The author would like to thank Wolfgang Ziller and Burkhard Wilking for sharing some insightful ideas that are incorporated into this paper. References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19]

I. BELEGRADEK and V. KAPOVITCH, Topological obstructions to nonnegative

curvature, Math. Ann. 320 (2001), 167 – 190. MR 2002d:53044 77 , Obstructions to nonnegative curvature and rational homotopy theory, preprint, arXiv:math.DG/0007007 77 A. L. BESSE, Einstein Manifolds, Ergeb. Math. Grenzgeb. (3) 10, Springer, Berlin, 1987. MR 88f:53087 83, 84, 89, 90 ´ L. M. CHAVES, A. DERDZINSKI, and A. RIGAS, A condition for positivity of curvature, Bol. Soc. Brasil. Mat. (N.S.) 23 (1992), 153 – 165. MR 93j:53050 79 J. CHEEGER and D. GROMOLL, On the structure of compete manifolds of nonnegative curvature, Ann. of Math. (2) 96 (1972), 413 – 443. MR 46:8121 77, 82, 83 ´ A. DERDZINSKI and A. RIGAS, Unflat connections in 3-sphere bundles over S 4 , Trans. Amer. Math. Soc. 265 (1981), 485 – 493. MR 82i:53026 80 K. GROVE and W. ZILLER, Curvature and symmetry of Milnor spheres, Ann. of Math (2) 152 (2000), 331 – 367. MR 2001i:53047 77 L. GUIJARRO, Improving the metric in an open manifold with nonnegative curvature, Proc. Amer. Math. Soc. 126 (1998), 1541 – 1545. MR 98j:53042 93, 98 , On the metric structure of open manifolds with nonnegative curvature, Pacific J. Math. 196 (2000), 429 – 444. MR 2002b:53042 83 L. GUIJARRO and G. WALSCHAP, The metric projection onto the soul, Trans. Amer. Math. Soc. 352 (2000), 55 – 69. MR 2000c:53034 81, 93 S. KRONWITH, Convex manifolds of nonnegative curvature, J. Differential Geom. 14 (1979), 621 – 628. MR 82k:53063 98 V. MARENICH, The holonomy in open manifolds of nonnegative curvature, Michigan Math. J. 43 (1996), 263 – 272. MR 97c:53054 88 ¨ M. OZAYDIN and G. WALSCHAP, Vector bundles with no soul, Proc. Amer. Math. Soc. 120 (1994), 565 – 567. MR 94d:53057 77 G. PERELMAN, Proof of the soul conjecture of Cheeger and Gromoll, J. Differential Geom. 40 (1994), 209 – 212. MR 95d:53037 82 M. STRAKE and G. WALSCHAP, 6-flat manifolds and Riemannian submersions, Manuscripta Math. 64 (1989), 213 – 226. MR 90g:53054 83, 84 , Connection metrics of nonnegative curvature on vector bundles, Manuscripta Math. 66 (1990), 309 – 318. MR 91a:53070 78, 79, 99 K. TAPP, Finiteness theorems for submersions and souls, Proc. Amer. Math. Soc. 130 (2002), 1809 – 1817. MR 2002m:53053 77 G. WALSCHAP, Soul-preserving submersions, Michigan Math. J. 41 (1994), 609 – 617. MR 95h:53042 84 A. WEINSTEIN, Fat bundles and symplectic manifolds, Adv. Math. 37 (1980), 239 – 250. MR 82a:53038 80

CONDITIONS FOR NONNEGATIVE CURVATURE

101

[20]

D. YANG, On complete metrics of nonnegative curvature on 2-plane bundles, Pacific J.

[21]

W. ZILLER, Fatness revisited, lecture notes, Univ. of Pennsylvania, Philadelphia, 1999.

Math. 171 (1995), 569 – 583. MR 96k:53034 77, 80 80

Department of Mathematics, Bryn Mawr College, Bryn Mawr, Pennsylvania 19010-2899, USA; [email protected]

TABLEAU ATOMS AND A NEW MACDONALD POSITIVITY CONJECTURE L. LAPOINTE, A. LASCOUX, and J. MORSE

Abstract Let 3 be the space of symmetric functions, and let Vk be the subspace spanned by the modified Schur functions {Sλ [X/(1−t)]}λ1 ≤k . We introduce a new family of symmetric (k) polynomials, {Aλ [X ; t]}λ1 ≤k , constructed from sums of tableaux using the charge (k) statistic. We conjecture that the polynomials Aλ [X ; t] form a basis for Vk and that the Macdonald polynomials indexed by partitions whose first part is not larger than k expand positively in terms of our polynomials. A proof of this conjecture would not only imply the Macdonald positivity conjecture, but also substantially refine it. (k) Our construction of the Aλ [X ; t] relies on the use of tableau combinatorics and yields various properties and conjectures on the nature of these polynomials. Another (k) important development following from our investigation is that the Aλ [X ; t] seem to play the same role for Vk as the Schur functions do for 3. In particular, this has led us to the discovery of many generalizations of properties held by the Schur functions, such as Pieri-type and Littlewood-Richardson-type coefficients. Contents 1. Introduction . . . . . . . . . . . . . 2. Background . . . . . . . . . . . . . 2.1. Symmetric function theory . . . 2.2. Tableau combinatorics . . . . . (k) 3. Definition of Aλ [X ; t] . . . . . . . 4. Main conjecture . . . . . . . . . . . 5. Embedded tableau decomposition . . 6. Irreducible atoms . . . . . . . . . . . 6.1. Cases k = 2 and k = 3 . . . . . 6.2. Generalized Kostka polynomials 7. The k-conjugation of a partition . . . 8. Pieri rules . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 1, Received 1 September 2000. Revision received 14 November 2001. 2000 Mathematics Subject Classification. Primary 05E05.

103

103 110 110 112 114 119 121 128 134 136 137 140

104

9. Hook case References . .

LAPOINTE, LASCOUX, and MORSE

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

141 144

1. Introduction We work with the algebra 3 of symmetric functions in the formal alphabet x1 , x2 , . . . with coefficients in Q(q, t). We use λ-ring notation in our presentation, and we refer those unfamiliar with this device to Section 2. It develops that the filtration of 3 given by the spaces Vk = {Sλ [X/(1 − t)]}λ1 ≤k with k ∈ N (1.1) provides a natural setting for the study of the (q, t)-Kostka coefficients K λµ (q, t). In fact, this filtration leads to a family of positivity conjectures refining the original Macdonald positivity conjecture, which now holds following the proof (see [4]) of the n! conjecture (see [2]). To see how this comes about, we first introduce some notation. We use a modification of the Macdonald integral forms Jµ [X ; q, t] that is obtained by setting X Hµ [X ; q, t] = Jµ [X/(1 − t); q, t] = K λµ (q, t)Sλ [X ]. (1.2) λ

The integral form Jµ [X ; q, t] at q = 0 reduces to the Hall-Littlewood polynomial Jµ [X ; 0, t] = Q µ [X ; t]. We also use a modification of Q µ [X ; t]: Hµ [X ; t] = Q µ [X/(1 − t); t] =

X

K λµ (t)Sλ [X ],

(1.3)

λ≥µ

where K λµ (t) is the Kostka-Foulkes polynomial. This given, we should note that bases for Vk also include the families (see [12]) {Hµ [X ; t]}µ1 ≤k

and

{Hµ [X ; q, t]}µ1 ≤k .

(1.4)

Our main contribution is the construction of a new family, (k)

{Aλ [X ; t]}λ1 ≤k ,

(1.5)

which we conjecture forms a basis for Vk and whose elements, in a sense that can be made precise, constitute the smallest Schur positive components of Vk . For this (k) reason we have chosen to call the Aλ [X ; t] the atoms of Vk . We begin by outlining the characterization of our atoms, which may be compared to the combinatorial construction of the Hall-Littlewood polynomials. The formal

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

105

sum, or the set, of all semistandard tableaux (hereafter called tableaux) with evaluation µ is denoted by Hµ ,∗ with the convention that H0 is the empty tableau. It was shown in [10] that X Hµ [X ; t] = z(Hµ ) = t charge(T ) Sshape(T ) [X ], (1.6) T ∈Hµ

where z is the functional z(T ) = t charge(T ) Sshape(T ) [X ].

(1.7)

The formal sum Hµ arises from a recursive application of promotion operators Br such that Br Hλ = H(r,λ) : Hµ = Bµ1 · · · Bµn−1 Bµn H0 .

(1.8)

The operators Br are tableau analogues of the operators building recursively the HallLittlewood polynomials presented in [5] and [3]. To construct the atoms of Vk , we introduce a family of filtering operators, Pλ→k , which have the effect of removing certain elements from the sum of tableaux in (1.8).∗ That is, given a k-bounded partition µ (a partition µ such that µ1 ≤ k), the atom (k) Aµ [X ; t] is X (k) A(k) t charge(T ) Sshape(T ) [X ] , (1.9) µ [X ; t] = z(Aµ ) = (k)

T ∈Aµ (k)

where Aµ is the formal sum of tableaux obtained from A(k) µ = Pµ→k Bµ1 · · · P(µn−1 ,µn )→k Bµn−1 P(µn )→k Bµn H0 . Following from this construction is the expansion X (k) (k) A(k) vλµ (t)Sλ [X ] with 0 ⊆ vλµ (t) ⊆ K λµ (t), µ [X ; t] = Sµ [X ] +

(1.10)

(1.11)

λ>µ

where for two polynomials P, Q ∈ Z[q, t] we write P ⊆ Q to mean Q− P ∈ N[q, t]. Originally, the atoms were empirically constructed with the idea that they could be characterized by (1.11) and the following two properties. (i) For k-bounded partitions λ and µ, and for any nonzero coefficient c(t) ∈ N[t], (k)

Aλ [X ; t] − c(t) A(k) µ [X ; t] 6= ∗ Blackboard ∗ The

X

vν (t) Sν [X ],

where vν ∈ N[t]. (1.12)

ν

fonts are used to distinguish sets of tableaux or operators on tableaux from functions. effect of the filtering operator is to extract λ-katabolizable tableaux (see Section 2.2 and [18]).

106

(ii)

LAPOINTE, LASCOUX, and MORSE

For any k-bounded partition µ, X (k) (k) (k) Hµ [X ; t] = A(k) K λµ (t) Aλ [X ; t] with K λµ (t) ∈ N[t]. µ [X ; t] + λ>µ

(1.13) However, our computer experimentation supported the following stronger conjecture, which connects the atoms to Macdonald polynomials indexed by k-bounded partitions: X (k) (k) Hµ [X ; q, t] = K λµ (q, t) Aλ [X ; t] (1.14) λ

with

(k)

0 ⊆ K λµ (q, t) ⊆ K λµ (q, t).

(1.15)

This has been the primary motivation for the research that led to this work. In particular, given the positive expansion in (1.11), property (1.14) with (1.15) would not only prove the Macdonald positivity conjecture, but also constitute a substantial strengthening of it. It transpires that the atoms are a natural generalization of the Schur functions. In (k) fact, our construction of Aλ [X ; t] yields the fact that for large k (k ≥ |λ|), (k)

Aλ [X ; t] = Sλ [X ].

(1.16)

Thus the atoms of 3 = V∞ are none other than the Schur functions themselves. (k) Moreover, computer exploration has revealed that the Aλ [X ; t] have a variety of remarkable properties extending and refining well-known properties of Schur functions. For example, we have observed generalizations of Pieri and Littlewood-Richardson rules, a k-analogue of the Young lattice induced by the multiplication action of e1 , and a k-analogue of partition conjugation. Further, we have noticed that the atoms satisfy, on any two alphabets X and Y , X (k) λ λ (k) (t) ∈ N[t]. where gµρ Aλ [X + Y ; t] = gµρ (t) A(k) µ [X ; t] Aρ [Y ; t], |µ|+|ρ|=|λ| λ (t) appearing here is a natural property of Schur The positivity of the coefficients gµρ functions not shared by the Hall-Littlewood or Macdonald functions. Finally, the atoms of Vk , when embedded in the atoms of Vk 0 for k 0 > k, seem to decompose positively: X (k→k 0 ) 0) (k) (k 0 ) (k→k 0 ) Aλ [X ; t] = Aλ [X ; t] + vµλ (t) A(k where vµλ (t) ∈ N(t). µ [X ; t], µ>λ

(1.17)

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

107

The tableau combinatorics involved in our construction and in hypothesis (1.13) suggests that the atoms provide a natural structure on the set of tableaux Hµ . For example, we have observed that for a k-bounded partition µ, Hµ decomposes into dis(k) joint subsets AT indexed by their element of minimal charge. Each of these subsets is characterized by the fact that its cyclage-cocyclage poset structure is isomorphic to (k) that of Ashape(T ) , and since (k)

(k)

z(AT ) = t charge(T ) Ashape(T ) [X ; t], (k)

(k)

(1.18)

(k)

we say that AT is a copy of Ashape(T ) . Therefore, if Cµ is the collection of tableaux indexing the copies that occur in the decomposition of Hµ , we have X (k) Hµ = AT . (1.19) (k)

T ∈C µ (k)

(k)

Note that the tableaux in AT have evaluation µ, while those in Ashape(T ) have, from (1.10), evaluation given by the shape of T . Now, (1.18) and (1.19) imply that the (k) coefficients K λµ (t) occurring in (1.13) are simply given by the formula (k)

K λµ (t) =

t charge(T ) .

X

(1.20)

(k) T ∈C µ

shape(T )=λ (k)

Since the promotion operators B` acting on AT produce collections of tableaux of the same evaluation, we examine their decomposition into copies as well. It appears that X (k) (k) B` AT = AT 0 , (1.21) (k)

T 0 ∈ET,` (k)

where ET,` is a suitable subcollection of the tableaux T 0 of shape ν such that ν/ shape(T ) is a horizontal `-strip. Therefore, formula (1.21) may be considered a refinement of the classical Pieri rules. In fact, letting t = 1 and shape(T ) = λ in (1.21), we have X (k) h ` [X ]Aλ [X ; 1] = A(k) (1.22) ν [X ; 1], (k)

ν∈E λ,` (k)

where E λ,` is a subset of the collection of shapes ν such that ν/λ is a horizontal ` (k) strip. We give a simple combinatorial procedure that we believe determines E λ,` . When ` = 1 in (1.22), we are led to a k-analogue of the Young lattice. This is the poset whose elements are k-bounded partitions and whose Hasse diagram is (k) obtained by linking an element λ to every µ ∈ E λ,1 . In Figure 1 we illustrate the

108

LAPOINTE, LASCOUX, and MORSE

poset obtained for degree 6 with k = 3. Moreover, the number of paths in this poset joining the empty partition to the partition λ is simply the number of summands in (k) (1.20) when µ = 1|λ| , namely, K λ,1|λ| (1). An analogous observation can be made for a general µ.

.

Figure 1. 3-analogue of the Young lattice

Central to our research is the observation that not all of the atoms need to be constructed using (1.10). In fact, for each k there is a distinguished “irreducible” subset of atoms of Vk from which all successive atoms may be constructed by simply applying certain generalized promotion operators. To be more precise, let a partition µ with no more than i parts equal to k − i be called k-irreducible (note: there are k! such partitions). Thus any k-bounded partition can be obtained by the partition rearrangement of the parts of a k-irreducible partition and a sequence of k-rectangles, partitions of the form (`k+1−` ) for ` = 1, . . . , k. This given, we let the collection of k-irreducible atoms be only the atoms indexed by k-irreducible partitions. This suggests that there are certain generalized promotion operators indexed by k-rectangles yielding that every atom may be written in the form (k)

Aλ [X ; t] = t c z(B R1 B R2 · · · B R` A(k) µ ),

(1.23)

where µ is a k-irreducible partition, R1 , . . . , R` are certain k-rectangles, and c ∈ N.

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

109

Again we find it interesting to consider the case t = 1. First, since the HallLittlewood polynomials at t = 1 are simply H(µ1 ,...,µn ) [X ; 1] = h µ1 [X ] · · · h µn [X ],

(1.24)

we see that Vk reduces to the polynomial ring Vk (1) = Q[h 1 , . . . , h k ]. Further, since the construction in (1.23) is simply multiplication by Schur functions when t = 1, (k)

Aλ [X ; 1] = S R1 [X ]S R2 [X ] · · · S R` [X ]A(k) µ [X ; 1],

(1.25)

it thus follows that k-irreducible atoms constitute a natural basis for the quotient Vk (1)/Ik , where Ik is the ideal generated by Schur functions indexed by krectangles. In fact, the irreducible atom basis offers a very beautiful way to carry out operations in this quotient ring. First work in Vk (1) using atoms, and then replace by zero all atoms indexed by partitions that are not k-irreducible. We examine our k-analogue of the Young lattice restricted to k-irreducible partitions. Figure 2 gives cases k = 3 and k = 4, where vertices denote irreducible atoms rather than partitions. Since it can be shown that the collection of monomials of the form {h 11 , h 22 , . . . , h kk }0≤i ≤k−i provides a basis for the quotient Vk (1)/Ik , it follows that the Hilbert series FVk (1)/Ik (q) of this quotient, as well as the rank generating function of the corresponding poset, is given by FVk (1)/Ik (q) =

k−1 Y

(1 + q i + q 2i + · · · + q (k−i)i ).

(1.26)

i=1

Finally, we make connections between our work and contemporary research in this area. We discovered that tableau manipulations identical to ours have been used for a different purpose in [16], [17], and [18]. In particular, certain cases of the generalized Kostka polynomials can be expressed in our notation as z(B R1 · · · B R` H0 ),

(1.27)

where R1 , . . . , R` is a sequence of rectangles whose concatenation is a partition (see [17]). When R1 , . . . , R` is a sequence of k-rectangles, this is simply the case µ = ∅ in (1.23). Thus it is again apparent that an integral part of our work lies in the kirreducible atoms, without which the atoms in general could not be constructed. Furthermore, it is known that these generalized Kostka polynomials can be built from the symmetric function operators B R introduced in [19]. The connection we have made with our atoms and these polynomials thus suggests that any atom can be obtained by applying a succession of operators B R indexed by k-rectangles to a given (k) irreducible atom Aµ [X ; t]: (k)

Aλ [X ; t] = t c B R1 B R2 · · · B R` A(k) µ [X ; t],

where c ∈ N.

(1.28)

110

LAPOINTE, LASCOUX, and MORSE

Level 3

Level 4

A(4) 0 ... .. ...... .

A(4) 1

...... ............ ...... ...... ......... ...... ........... ................. .............

A(3) 0 .. ...... ...... . ...... .......... .............

(3) A .. ... .1

...... ...... ...... . .......... ...........

.... ...... ...... ............ .............

A(4) 2...

A(3) 1,1

A(3) 2,1 ... . ...... .

(3)

A2,1,1

A1,1 ....

... ... ...... .

A(4) 3,2

(4)

... ... ...... .

(4) A2,2 ........ .................

..... ...... ... ..... ............................. ....... ...... .... .......................... .. ........... .......... ...................... ......... ................

A(4) 3,2,1... A(4) 3,2,2

...... ...... ...... .......... ................

(4)

... ... ...... .

(4)

A2,1,1

. ...... ...... ...... ........... ...........

...... ...... ...... .......... ................

A(4) 2,1,1,1

............ .. .............. ...... .. .............. ...... ...... ... .................... ...... ........... ............................. . . . . . . . . . . . . . . . . . . . . . . . ........... .......... .......................

A(4) 3,1,1,1

....... ....... ....... ....... ........... ................

A1,1,1

A(4) 2,2,1

. .. ....... ............ ...... ...... . . ...... .......... .......... ............. ...........

... .. ...... .

(4)

....... ....... ....... ....... ........... ...............

...... ...... ...... .......... ................

A(4) 3,1,1

...... ...... ....... .......... ................

...... ...... ....... . ...................

A2,1 ....

...... ....... ....... ....... ............. . . . ..............

(4) A3,1

....... ...... ....... ........... .............

...... ...... ....... .. ...................

(4)

A3

. ...... ...... ...... ........... .............

...... ...... ...... . .......... ............

(4)

(4)

A2....

... .. ...... .

... .. ...... .

.. ...... ....... ....... ...... . . . . . . . . ....... .............

A(4) 3,2,1,1

. ....... ...... . ...... ........... ...........

A3,2,2,1 ....

A(4) 2,2,1,1 ... .. ...... .

A(4) 2,2,1,1,1

...... ... ...... ...... ...... ...... .......... . ...... ................. .....................

(4)

A3,2,1,1,1 ....

...... ...... ....... .. ....................

.. ...... ....... ............ .............

A(4) 3,2,2,1,1 ... .. ...... .

A(4) 3,2,2,1,1,1 Figure 2. Action of e1 on irreducible atoms of level 3 and level 4

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

111

Note that this is a symmetric function analogue of (1.23) and specializes to (1.25) when t = 1.

2. Background 2.1. Symmetric function theory Here, symmetric functions are indexed by partitions, sequences of nonnegative integers λ = (λ1 , λ2 , . . .) with λ1 ≥ λ2 ≥ · · · . The order of λ is |λ| = λ1 + λ2 + · · · , P the number of nonzero parts in λ is denoted `(λ), and n(λ) = i (i − 1)λi . We use the dominance order on partitions with |λ| = |µ|, where λ ≤ µ when λ1 + · · · + λi ≤ µ1 + · · · + µi for all i. For two partitions λ and µ, λ ∪ µ denotes the partition rearrangement of the parts of λ and µ. Every partition λ may be associated to a Ferrers diagram with λi lattice squares in the ith row, from the bottom to top. For example, λ = (4, 3, 1) =

(2.1)

For each cell s = (i, j) in the diagram of λ, let `0 (s), `(s), a(s), and a 0 (s) be, respectively, the number of cells in the diagram of λ to the south, north, east, and west of the cell s. The transposition of a diagram associated to λ with respect to the main diagonal gives the conjugate partition λ0 . For example, the conjugate of (4,3,1) is λ0 = = (3, 2, 2, 1). (2.2) A skew diagram µ/λ, for any partition µ containing the partition λ, is the diagram obtained by deleting the cells of λ from µ. The thick frames below represent (5,3,2,1)/(4,2): (2.3) We employ the notation of λ-rings, needing only the formal ring of symmetric functions 3 to act on the ring of rational functions in x1 , . . . , x N , q, t, with coefficients in Q. The action of a power sum pi on a rational function is, by definition, h P c u i P c ui α α α α pi Pα = Pα i d v β β β β dβ vβ

(2.4)

with cα , dβ ∈ Q and u α , vβ monomials in x1 , . . . , x N , q, t. Since the ring 3 is generated by power sums pi , any symmetric function has a unique expression in terms of pi , and (2.4) extends to an action of 3 on rational functions. In particular, f [X ], the action of a symmetric function f on the polynomial X = x1 + · · · + x N , is simply f (x1 , . . . , x N ).

112

LAPOINTE, LASCOUX, and MORSE

We recall that the Macdonald scalar product, h , iq,t , on 3 ⊗ Q(q, t) is defined by setting

`(λ) h1 − q i Y 1 − q λi  pλ [X ], pµ [X ] q,t = δλµ z λ = δ z p , λµ λ λ 1 − t λi 1−t

(2.5)

i=1

where for a partition λ with m i (λ) parts equal to i we associate the number z λ = 1m 1 m 1 ! 2m 2 m 2 ! · · · .

(2.6)

The Macdonald integral forms Jλ [X ; q, t] are then uniquely characterized (see [12]) by (i) hJλ , Jµ iq,t = 0 if λ 6= µ, X (ii) Jλ [X ; q, t] = vλµ (q, t)Sµ [X ],

(2.7) (2.8)

µ≤λ

(iii) vλλ (q, t) =

Y (1 − q a(s) t `(s)+1 ),

(2.9)

s∈λ

where Sµ [X ] is the usual Schur function and vλµ (q, t) ∈ Q(q, t). 2.2. Tableau combinatorics A ∗ denotes the free monoid generated by the alphabet A = {1, 2, . . . }, and Q[A ∗ ] is the free algebra of A . Elements of A ∗ are called words, and for E a subset of A , w E denotes the subword obtained by removing from w all the letters not in E. The degree of a word w is denoted |w|, and if w has ρ1 ones, ρ2 twos, and so on, and ρm m’s, then the evaluation of w is (ρ1 , . . . , ρm ). For example, w = 131332 has degree 6 and evaluation (2,1,3). A word w of degree n is standard if and only if its evaluation is (1, . . . , 1). Recall that a word w is Yamanouchi in the letters a1 < · · · < ah if it is such that for every factoring w = uv, v contains more ai than a j for all i < j. The plactic monoid on the alphabet A is the quotient A ∗ / ≡, where ≡ is the congruence generated by the Knuth relations (see [6]) defined on three letters a, b, c by a c b ≡ c a b,

a ≤ b < c,

b a c ≡ b c a,

a < b ≤ c.

(2.10)

Two words w and w0 are said to be Knuth equivalent if and only if w ≡ w0 . In this paper, a tableau is a filling of a Ferrers diagram with positive integer entries that are nondecreasing in rows and increasing in columns: T =

6 7 4 4 5 1 1 1 2 3

(2.11)

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

113

The word w obtained by reading the entries of a tableau from left to right and top to bottom is said to be a tableau word, or simply a tableau. Our example shows that w = 6744511123 is a tableau with evaluation (3,1,1,2,1,1,1). A standard tableau T is a tableau of evaluation (1, 1, . . . , 1). For example, T =

7 4 6 1 2 3 5

(2.12)

The transpose of a standard tableau T t is defined in the same manner as the transpose of a Ferrers diagram. With T as given in (2.12), we have Tt =

5 3 2 6 1 4 7

(2.13)

Since the transpose of a tableau is assured to be a tableau only when the original tableau is standard, this definition is valid only for standard tableaux. We assume that readers are familiar with the Robinson-Schensted correspondence (see [13], [14])
w ←→ P(w), Q(w) (2.14) providing a bijection between a word w and a pair of tableaux (P(w), Q(w)), where P(w) is the only tableau Knuth equivalent to w and Q(w) is a standard tableau. The ring of symmetric functions is embedded into the plactic algebra by sending the Schur function Sλ to the sum of all tableaux of shape λ (see [1], [9]). The commutativity of the product Sλ Sµ ≡ Sµ Sλ thus implies bijections among tableaux. In particular, we can define the following action of the symmetric group on words (see [11]). The elementary transposition σi permutes degrees in i and i + 1. Given a word w of evaluation (ρ1 , . . . , ρm ), let u denote the subword in letters a = i and b = i + 1. The action of the transposition σi affects only the subword u and is defined as follows. Pair every factor b a of u, and let u 1 be the subword of u made out of the unpaired letters. Pair every factor b a of u 1 , and let u 2 be the subword made out of the unpaired letters. Continue in this fashion as long as possible. When all factors b a are paired and unpaired letters of u are of the form a r bs , σi sends a r bs → a s br . For instance, to obtain the action of σ2 on w = 123343222423, we have u = w{2,3} = 233322223, and the pairings are
2 3(3(32)2)2 23, (2.15) which means that σ2 u = 233322233 and σ2 w = 123343222433. It is verified in [11] that the σi ’s obey the Coxeter relations and thus provide an action of the symmetric group on words. We use the notion of charge (see [9], [10]) defined by writing a word w, with evaluation given by a partition, counterclockwise on a circle with * separating the

114

LAPOINTE, LASCOUX, and MORSE

end of the word from its beginning and then summing the labels that are obtained by the following procedure. Let ` = 0. Moving clockwise from *, we label with ` the first occurrence of letter 1, then the first occurrence of letter 2 following this 1, then the first occurrence of letter 3 following this 2, and so on, with the condition that each time the * is passed, the label ` is increased to ` + 1. Once each of the letters 1, 2, 3, . . . have been labeled, we repeat this procedure on the unlabeled letters, again starting at the * with ` = 0. The process ends when all letters have been labeled. We can define charge on a word w whose evaluation is not a partition by first permuting the evaluation to a partition using σ and then taking the charge of σ w. Figure 3 shows that charge(12114123234) = 0+0+0+0+1+0+1+1+1+1+2 = 7.

∗ 4

1

0

2

0

2

3

1

0

1

2

1

0

1

3 1

4

0

2

1

1

1

Figure 3. Charge of w = 12114123234

The definition of charge means that a tableau of shape and evaluation µ has charge zero, and thus the combinatorial construction for the Hall-Littlewood polynomials (1.6) implies X Hµ [X ; t] = Sµ [X ] + K λµ (t)Sλ [X ]. (2.16) λ>µ (k)

3. Definition of Aλ [X ; t] Our main contribution is the method for constructing new families of functions whose significance has been outlined in the introduction. The characterization is similar to the combinatorial definition of Hall-Littlewood polynomials using the set of tableaux that arises from a recursive application of promotion operators. Our families also correspond to a set of tableaux generated by the promotion operators, but here we introduce new operators Pλ→k to eliminate undesirable elements. To be precise, we now define the operators involved in our construction. The promotion operators are defined on a tableau T with evaluation (λ1 , . . . , λm )

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

115

by Br (T ) = σ1 · · · σm Rr T,

(3.1)

where Rr adds a horizontal r -strip of the letter m + 1 to T in all possible ways. For example, B3

2 2 1 1

= σ1 σ2 R3 21 21  = σ1 σ2 12 21 3 3

3

3 2 2 1 1 3 3

+

+

3 3 2 2 1 1 3



=

2 2 1 1 1 3 3

+

3 2 2 1 1 1 3

+

3 3 2 2 1 1 1

Note that the action of σ implies that the resulting tableaux have evaluation (r, λ1 , . . . , λm ). While our construction relies on these operators, they generate certain unwanted tableaux. We now present the concepts needed to obtain operators that filter out such elements. The main hook length of a partition λ, h M (λ), is the hook length of the cell s = (1, 1) in the diagram associated to λ. That is, h M (λ) = `(s) + a(s) + 1 = λ1 + λ01 − 1 = λ1 + `(λ) − 1.

(3.2)

For example, if λ = (4, 3, 1), then h M (λ) =

• • • • • •

= 6.

(3.3)

Any k-bounded partition λ can be associated to a sequence of partitions called the ksplit, λ→k = (λ(1) , λ(2) , . . . , λ(r ) ). The k-split of λ is obtained by dividing λ (without changing the order of its entries) into partitions λ(i) where h M (λ(i) ) = k, ∀ i 6= r . For example, (3, 2, 2, 2, 1, 1)→3 = ((3), (2, 2), (2, 1), (1)). Equivalently, we horizontally cut the diagram of λ into partitions λ(i) where h M (λ(i) ) = k. In our example this gives

(3.4) −→ Note that the last partition in the sequence λ→k may have main hook length less than k. As k increases, the k-split of λ contains fewer partitions. For k = 4,

−→

or


(3, 2, 2, 2, 1, 1)→4 = (3, 2), (2, 2, 1), (1) .

(3.5)

When k is big enough (h M (λ) ≤ k), then λ→k = (λ); that is, (3, 2, 2, 2, 1, 1)→8 = ((3, 2, 2, 2, 1, 1)).

116

LAPOINTE, LASCOUX, and MORSE

Let T be a given tableau whose shape contains λ. We denote by Tλ the subtableau of T of shape λ. Let U be the skew tableau obtained by removing Tλ from T , let T1 be the tableau contained in the first `(λ) rows of U , and let T2 be the portion of U that is above the `(λ) rows. Let us denote by T1 T2 the skew tableau obtained by juxtaposing T1 to the northwest corner of T2 , and by T the unique tableau that is Knuth equivalent to T1 T2 . For instance, in the figure below, λ = (3, 2, 1, 1), the skew tableau with empty cells is T1 , the tableau with bullets is T2 , the middle diagram is T1 T2 , and the right diagram is a possible shape for T : • • • • • • •

−→

(3.6)

≡ • • • • • • •

This construction permits us to define an operation on tableau Kλ called λ-katabolism: ( T if λ ⊆ shape(T ), Kλ (T ) = 0 otherwise. For example, the (2, 1)-katabolism of T = 9472581236 is

K(2,1)

9 4 7 2 5 8 1 2 3 6

−→

5 8 3 6

8 5 6 9 3 4 7

≡

9 4 7

(3.7)

Note that λ-katabolism was also introduced in [18] and, for the case where λ is a row, in [11]. Let S(λ) be the set of λ-shaped tableaux with evaluations (0m , λ1 , λ2 , . . .) for m ∈ N. For λ = (3, 2, 2), we have   n o 3 3 5 5 6 6 4 4 S = 2 2 , 3 3 , 4 4 , 5 5 , ... . (3.8) 1 1 1

2 2 2

3 3 3

4 4 4

This given, the restricted λ-katabolism Kλ is defined by setting ( Kλ (T ) if Tλ ∈ S(λ), Kλ (T ) = 0 otherwise.

(3.9)

For example, K(2,1) on the tableau in (3.7) is zero, whereas

K(2,1)

9 4 7 2 5 8 1 1 3 6

=

5 8 3 6 9 4 7

≡

8 5 6 9 3 4 7

(3.10)

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

117

For a sequence of partitions S = (λ(1) , λ(2) , . . . , λ(`) ), we define the filtering operator P S using the succession of restricted katabolisms Kλ(`) · · · Kλ(1) , ( T if Kλ(`) · · · Kλ(1) (T ) = H0 , P S (T ) = (3.11) 0 otherwise. In fact, we consider only the case where S is the sequence of partitions given by λ→k . PROPERTY 1 The filtering operators Pλ→k satisfy the following properties. (a) For T ∈ S(λ), we have Pλ→k T = T for all k such that λ is bounded by k. (b) For U a tableau of |λ| letters such that U 6∈ S(λ), Pλ→k U = 0 for all k ≥ h M (λ).

Proof By definition (3.11), we must show Kλ(`) · · · Kλ(1) T = H0 for λ→k = (λ(1) , . . . , λ(`) ). Recall that Kλ(1) T acts by extracting the bottom `(λ(1) ) rows of T and inserting into the remainder any entries not in Tλ(1) ∈ S(λ(1) ). Since the bottom `(λ(1) ) rows of T ∈ S(λ) are exactly Tλ(1) ∈ S(λ(1) ), the katabolism Kλ(1) T simply removes the bottom `(λ(1) ) rows of T . By iteration, we obtain the empty tableau. For (b), the condition that k is large implies that λ→k = (λ). It thus suffices to show that Kλ (U ) = 0. Now Kλ acts first by extracting from U the subtableau Uλ ∈ S(λ). If U is of shape λ, then Uλ = U 6∈ S(λ). If U is not of shape λ, since U has degree |λ|, then Uλ does not exist. Therefore we have our claim. These filtering operators are those required in the characterization of our families of functions. We thus have the tools to recursively define the central object in our work, (k) the super atom of shape λ and level k, Aλ . Definition 2 (k) Let A0 be the empty tableau. The super atom of a k-bounded partition λ is (k)

(k)

Aλ = Pλ→k Bλ1 (A(λ2 ,λ3 ,... ) ).

(3.12)

For example, given that we know the super atom (3)

A1,1,1,1 =

4 3 2 1

+

3 2 1 4

(3.13)

118

LAPOINTE, LASCOUX, and MORSE (3)

we can obtain A2,1,1,1,1 by first acting with the rectangular operator B2 on (3.13), (3)

B2 (A1,1,1,1 ) = B2



4 3 2 1

+

3 2 1 4



=

3 2 1 1 4 5

+

3 2 5 1 1 4

+

5 3 2 1 1 4

+

4 3 2 5 1 1

+

4 3 2 1 1 5

+

5 4 3 2 1 1

(3.14)

and then applying to these tableaux the operator Pλ→3 , where λ→3 = ((2, 1), (1, 1, 1)): (3)

(3)

A2,1,1,1,1 = P((2,1),(1,1,1)) (B2 A1,1,1,1 ) =

3 2 5 1 1 4

+

4 3 2 5 1 1

+

4 3 2 1 1 5

+

5 4 3 2 1 1

(3.15)

Our method for constructing the super atoms allows the derivation of several natural properties. In particular, these properties generally arise as the consequence of those held by the promotion and filtering operators. 3 For all k-bounded partitions λ, we have PROPERTY

(k)

Aλ ⊆ Hλ .

(3.16)

Proof For λ = (λ1 , . . . , λm ), recall from (1.8) that Hλ = Bλ1 · · · Bλm H0 .

(3.17)

On the other hand, following from Definition 2, we have (k)

(k)

Aλ = Pλ→k Bλ1 · · · P(λm )→k Bλm A0 .

(3.18)

(k)

Since Aλ is distinguished from Hλ only by acting with a filtering operator after each (k) application of a B` operator, we have that every tableau in Aλ is also in Hλ . PROPERTY 4 Let T be the tableau of shape and evaluation λ. The super atoms satisfy the following. (k) (i) T ∈ Aλ for any k such that λ is bounded by k. (k) (ii) Aλ = T for k ≥ h M (λ).

Proof (k) (k) (k) (i) Recall that Aλ = Pλ→k Bλ1 Aλ2 ,...,λ` . Assume by induction that U ∈ Aλ2 ,...,λ` where U has shape and evaluation (λ2 , . . . , λ` ). Rλ1 U produces a sum of tableaux,

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

119

one being the tableau of shape (λ1 , λ2 , . . . , λ` ), which is then sent to the tableau T of shape and evaluation λ under the action of the symmetric group. It thus suffices to show that T is not eliminated by Pλ→k for all k. This is shown in Property 1(a). (k) (ii) In particular, (i) implies that T ∈ Aλ for k ≥ h M (λ). By the definition (k) of Aλ , it thus suffices to show that Pλ→k U = 0 for all U 6= T . This is true by Property 1(b). As with the definition of the Hall-Littlewood polynomials, we associate symmetric functions to our super atoms. Definition 5 With z as in (1.7), we define the symmetric function atoms by (k)

(k)

Aλ [X ; t] = z(Aλ ).

(3.19)

Properties we have given for the super atoms allow us to deduce several properties of these functions. For example, an immediate consequence of Property 4(ii) is the following. PROPERTY 6 When k is large ( k ≥ h M (λ)), we have (k)

Aλ [X ; t] = Sλ [X ].

PROPERTY 7 The atoms are linearly independent and have an expansion of the form X (k) (k) (k) Aλ [X ; t] = Sλ [X ] + vµλ (t) Sµ [X ], where vµλ (t) ∈ N[t].

(3.20)

(3.21)

µ>λ

Proof (k) (k) We have shown that Aλ ⊆ Hλ . Thus, by Definition 5, the triangularity of Aλ [X ; t] follows from the triangularity of Hλ [X ; t] (see (2.16)). Further, Property 4 implies that (k) the tableau T of shape and evaluation λ occurs in Aλ , and therefore z(T ) = Sλ [X ] (k) occurs in Aλ (T has charge zero).

120

LAPOINTE, LASCOUX, and MORSE

4. Main conjecture Our work to characterize the atoms was originally motivated by the belief that these polynomials play an important role in understanding the q, t-Kostka coefficients. More precisely, we have the following. 8 For any partition λ bounded by k, X (k) Hλ [X ; q, t] = K µλ (q, t) A(k) µ [X ; t], CONJECTURE

(k)

where K µλ (q, t) ∈ N[q, t].

µ;µ1 ≤k

(4.1)

For example, we have (2)

(2)

(2)

H2,1,1 [X ; q, t] = t A2,2 [X ; t] + (1 + qt 2 ) A2,1,1 [X ; t] + q A1,1,1,1 [X ; t] (3)

(3)

= t 2 A3,1 [X ; t] + (t + qt 2 ) A2,2 [X ; t] (3)

(3)

+ (1 + qt 2 )A2,1,1 [X ; t] + q A1,1,1,1 [X ; t] (k≥4)

= t 3 A4

(k≥4)

[X ; t] + (t + t 2 + qt 3 ) A3,1 [X ; t] (k≥4)

+ (t + qt 2 ) A2,2 [X ; t] (k≥4)

(k≥4)

+ (1 + qt + qt 2 )A2,1,1 [X ; t] + q A1,1,1,1 [X ; t].

(4.2)

This conjecture implies that the atoms of level k form a basis for Vk . Further, since the atoms expand positively in terms of Schur functions (3.21), our conjecture also implies Macdonald’s positivity conjecture on the Hλ [X ; q, t] in Vk . Since Property 6 gives (k) K µλ (q, t) = K µλ (q, t) for k ≥ |λ|, (4.3) we see that this conjecture is a generalization of Macdonald’s conjecture. In fact, our conjecture refines the original Macdonald conjecture in the following sense: substituting (3.21), the positive Schur function expansion of atoms, into (4.1), we have X (k) X (k) Hλ [X ; q, t] = K µλ (q, t) vνµ (t) Sν [X ]. (4.4) µ

ν≥µ

On the other hand, since the q, t-Kostka coefficients appear in the expansion X Hλ [X ; q, t] = K νλ (q, t) Sν [X ], ν

(4.5)

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

121

we have K νλ (q, t) =

X µ≤ν

(k)

(k) K µλ (q, t) vνµ (t)

(k)

= K νλ (q, t) +

X µ<ν

(k)

(k) K µλ (q, t) vνµ (t).

(4.6)

(k)

Since vνµ (t) is in N[q, t], Conjecture 8 implies that (k)

K µλ (1, 1) ≤ K µλ (1, 1),

(4.7)

where K µλ (1, 1) is known to be the number of standard tableaux of shape µ. Thus the problem of finding a combinatorial interpretation for the K µλ (q, t) coefficients (i.e., associating statistics to standard tableaux) is reduced to obtaining statistics for (k) the fewer K µλ (q, t). Based on our conjecture, we have the following corollary concerning the expansion of Hall-Littlewood polynomials in terms of our atoms. COROLLARY 9 Assuming that Conjecture 8 holds, we have, for any partition λ bounded by k, X (k) (k) Hλ [X ; t] = K µλ (t) A(k) where K µλ (t) ∈ N[t]. (4.8) µ [X ; t], µ≥λ

If we consider this corollary as the result of applying z to an identity on tableaux X (k) z (Hλ ) = K µλ (t) z(A(k) where K µλ (t) ∈ N[t], (4.9) µ ), µ

then it suggests that the set of all tableaux with evaluation λ can naturally be decom(k) (k) posed into subsets that are mapped under z to the atoms Aµ [X ; t]. Here, K µλ (1) corresponds to the number of times such a subset occurs in Hλ , which, by (4.7), is such that (k) K µλ (1) ≤ K µλ (1) , (4.10) where K µλ (1) is the number of tableaux with evaluation λ and shape µ. These subsets (k) are called copies of Aµ , and they provide a natural decomposition for the set of tableaux of a given evaluation. 5. Embedded tableau decomposition We expect from (4.9) that the set of all tableaux with given evaluation can be decomposed into subsets associated to our super atoms. These subsets are characterized by a cyclage-cocyclage ranked-poset structure (see [11]).

122

LAPOINTE, LASCOUX, and MORSE

For tableau T = xw where x is not the smallest letter of T , we define T 0 to be the unique tableau such that T 0 ≡ wx. The mapping T → T 0 is a called a cyclage and is such that charge(T 0 ) = charge(T ) + 1 if the evaluation of T is a partition. For tableau T = wx where x is not the smallest letter of T , we define T 0 to be the unique tableau such that T 0 ≡ xw. The cocyclage is the mapping T → T 0 and is such that charge(T 0 ) = charge(T ) − 1 if the evaluation of T is a partition. On any collection of tableaux T of the same evaluation, we can define a poset (T,
3 2 2 5 1 1 1 3 4

% . 2

4 3 2 2 5 1 1 1 3

1

4 3 3 2 2 1 1 1 5

-

% &

l

l

4 3 3 2 2 5 1 1 1

5 4 3 2 2 1 1 1 3

& 0

4 3 2 2 1 1 1 3 5

3 3 2 2 5 1 1 1 4

(5.1)

. 5 4 3 3 2 2 1 1 1

where the arrows indicate the cyclage and cocyclage relations between tableaux. CONJECTURE 10 The cyclage and cocyclage induce a connected ranked-poset structure on the set of (k) tableaux contained in a given super atom Aλ .

Given a collection of tableaux T, the Hasse diagram of the poset (T,
TABLEAU ATOMS AND A POSITIVITY CONJECTURE

123

(4)

A3,2,2,1,1 , is charge c+3 



c+2  

|

|

(5.2)

c+1 



c We can now define the subsets associated to our atoms. Definition 11 If a set of tableaux T has the properties (1) T is the tableau of minimal charge in T, (k) (2) 0T = 0shape(T ) , (k)

(k)

then this set is called a copy of the atom Ashape(T ) and is denoted AT . This given, if the posets are connected (Conjecture 10), then the charges associated to (k) (k) the elements of a super atom Aλ differ from those of a copy atom AT by a common (k) factor. Furthermore, since there is a unique element of zero charge in Aλ (the tableau (k) with shape and evaluation λ), then it is the minimal element in AT and we have (k)

(k)

z(AT ) = t charge(T ) Aλ [X ; t], (k)

where shape(T ) = λ.

(5.3) (k)

Note also that the tableaux in Aλ have evaluation λ, while those in AT have eval(k) (k) uation given by the evaluation of T . That is, Aλ = AT only if T is of shape and

124

LAPOINTE, LASCOUX, and MORSE (4)

(4)

evaluation λ. The copies of A3,2,2,1,1 include A863925147 , given by charge 13

3 2 5 7 1 4 6 8 9

% . 12

9 3 2 5 7 1 4 6 8

11

6 3 9 2 5 1 4 7 8

6 3 2 5 1 4 7 8 9

3 9 2 5 7 1 4 6 8

% &

l

l

8 3 9 2 5 7 1 4 6

9 6 3 2 5 1 4 7 8

&

(5.4)

. 8 6 3 9 2 5 1 4 7

10

It appears that there is a unique way to decompose the set of all tableaux Hµ (k) into atoms of level k ≥ µ1 . More precisely, we let Cµ denote the collection of all (k) tableaux T with evaluation µ where AT is a copy of a super atom of level k. Then we have the following. CONJECTURE 12 For any partition µ bounded by k, we have X (k) Hµ = AT .

(5.5)

(k) T ∈C µ

From Corollary 9 and (5.3), an implication of this identity under the mapping z is the following. 13 The k-Kostka-Foulkes polynomials are simply X (k) K λµ (t) = t charge(T ) . COROLLARY

(5.6)

(k) T ∈C µ

shape(T )=λ (k)

One method of obtaining the set Cµ is as follows. The element of minimal charge in (k) Hµ has shape µ and is thus also the minimal element of Aµ . Remove from Hµ all (k) tableaux in Aµ . Choose a tableau T with minimal charge from those that remain. T (k) must index a copy of the atom Aλ , where λ = shape(T ). From the Hasse diagram

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

125

(k)

(k)

0λ , it is possible to find and remove all tableaux in the atom AT . Repeat this procedure always on an element of minimal charge in the resulting sets. The collection (k) of these minimal elements is Cµ . Evidence suggests that this method also provides a direct decomposition of any copy atom of a level k into copy atoms of level k 0 > k. CONJECTURE 14 (k) For any atom AT such that shape(T ) is bounded by k and for any k 0 > k, (k)

(k 0 )

X

AT =

(5.7)

AT 0

(k→k 0 ) T 0 ∈DT

(k→k 0 )

for some collection of tableaux DT

.

On the level of functions, this translates into the following. COROLLARY 15 If λ is a partition bounded by k, and k 0 > k, then X (k→k 0 ) 0) (k) (k 0 ) Aλ [X ; t] = Aλ [X ; t] + vµλ (t) A(k µ [X ; t] , µ>λ

(k→k 0 )

where vµλ

(t) ∈ N[t]. (5.8)

This conjecture is a generalization of the result presented in Property 7 since we (k→k 0 ) recover vµλ (t) = vµλ (t) when k 0 ≥ |λ|. For examples that support the preceding conjectures, refer to Figures 4 and 5. These figures also suggest that the number of elements in an atom, at increasing charges, forms a unimodal sequence. Since an atom has a unique minimal element, these sequences always start with 1. CONJECTURE 16 Given any atom

(k)

Aλ [X ; t] =

X µ≥λ

the numbers #i =

X µ≥λ

(k)

vµλ (t) Sµ [X ],

(k) vµλ (t) i t

are such that [#0 , #1 , . . . ] is a unimodal sequence.

(5.9)

(5.10)

126

LAPOINTE, LASCOUX, and MORSE

Degree 3

Degree 4

1 1

2

2

3

4

3 4 1

3 1

2

3

2

2 1

3

4

3

1

2

1

2

3

2

2

1

4 3

3

4

1

1

4 2 1

3

4

1

3

2 1

4

4 3

LEVEL 3:

2

2

3

LEVEL 2:

4

3

2 1

Figure 4. Atomic decomposition of standard tableaux of degrees 3 and 4. In order to read the decomposition of a given level k, one must consider the lines associated to all the levels that are not bigger than k. That is, when doing the decomposition from one level to the other, lines are added without ever being removed. Thus, for instance, the tableaux 2413 and 3214 are in the same atom up to level 2 and in different atoms for any higher levels.

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

1 2

3

4

127

5

5 1

2 3

4

4 1 3

4

1

2

2 3

1

5

5

5

1

2 3 3

1

2

3

1

2

1

3

4

3

5

1

2 4

4

5

2

5

2

3

1

3 4

1

3

2 4

5

2

4

1

4

4

5 3

5

4

3

5

1

2 4

4 3 5 2

5

1 2 5

4 2 1

4

2

4

2 5

3

1

3 5

1 3

1

4 3

5

2

5

3

5

4

2

2 4

2 1

1

4 5

1 3

3

5 3

3

2 5

4

2

1 4

3

1

4

2 1

5 5 4

LEVEL 2: LEVEL 3: LEVEL 4:

5

3 2 1

Figure 5. Atomic decomposition of standard tableaux of degree 5. See Figure 4 for details on how to read the figure.

128

LAPOINTE, LASCOUX, and MORSE (4)

For example, the unimodal sequence associated to A3,2,2,1,1,1 [X ; t] is [1, 3, 5, 5, 3, 1]: (4)

A3,2,2,1,1,1 [X ; t] = S3,2,2,1,1,1 + t S4,2,1,1,1,1 + t S3,3,2,1,1 + (t + t 2 )S4,2,2,1,1 + t 2 S3,3,3,1 + t 2 S4,3,1,1,1 + (t 2 + t 3 )S5,2,1,1,1 + (t 2 + t 3 )S4,3,2,1 + t 3 S5,2,2,1 + t 3 S4,3,3 + (t 3 + t 4 )S5,3,1,1 + t 4 S6,2,1,1 + t 4 S5,3,2 + t 5 S6,3,1 .

(5.11)

We see later (Corollary 37 of Conjecture 36) that these sequences also seem to end with 1; that is, an atom has a unique element of maximal charge. We also provide a way to obtain the shape of this maximal element. Note that the sequences are not necessarily symmetric. For instance, from Figure 5 on page 127, we see that the sequence (2) associated to A1,1,1,1,1 [X ; t] is [1, 1, 2, 2, 1]. We finish this section by stating a conjecture that reiterates the importance of the atoms as a natural basis for Vk . CONJECTURE 17 For any two alphabets X and Y , (k)

Aλ [X + Y ; t] =

X

λ (k) gµρ (t) A(k) µ [X ; t] Aρ [Y ; t]

(5.12)

|µ|+|ρ|=|λ| λ (t) ∈ N[t]. with gµρ λ (t) appearing here It is important to note that the positivity of the coefficients gµρ is a natural property of Schur functions that is not shared by the Hall-Littlewood or Macdonald functions.

6. Irreducible atoms We have now seen that the super atoms can be constructed by generating sets of tableaux with promotion operators Br and then eliminating undesirable elements using the projection operators Pλ→k . Further, we have given a method for obtaining copies of the super atoms, allowing us to decompose the set of all tableaux with a (k) given evaluation and to provide natural properties on the functions Aλ [X ; t]. Remarkably, it appears that there is a method for constructing many of the atoms without generating any undesirable elements. In fact, what could be seen as the “DNA” of our atoms is a subset of irreducible atoms for each Vk , from which all successive atoms of Vk may be obtained by simply applying a generalized version of the promotion operators.

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

129

To be more precise, let a rectangular partition of the form (`k+1−` ) be referred to as a k-rectangle, and let a partition with no more than i parts equal to k − i be called k-irreducible. Definition 18 The collection of k-irreducible atoms is composed of atoms indexed by k-irreducible partitions. If an atom is not irreducible, then it is said to be reducible. 19 There are k! distinct k-irreducible partitions. PROPERTY

Proof A partition λ is k-irreducible if and only if λ has no more than i parts equal to k − i. There are obviously k! such partitions. The irreducible atoms of levels 1, 2, and 3 are (1)

k=1:

A0 ,

k=2:

A0 ,

(2)

A1 ,

k=3:

A0 ,

(3)

A1 ,

(2)

(3)

(3)

A2 ,

(3)

A1,1 ,

(3)

A2,1 ,

(3)

A2,1,1 .

(6.1)

Any k-bounded partition µ is of the form µ = λ ∪ R1 ∪ · · · ∪ Rn , where λ is a k-irreducible partition and R1 , . . . , Rn is a sequence of k-rectangles. In fact, we conjecture that any atom of level k can be obtained from a k-irreducible atom by the application of certain generalized promotion operators that are indexed by k-rectangles. Before we can introduce these promotion operators, we need to define an opera(h) tion that generalizes σi . We define σi to send a word w of evaluation (ρ1 , ρ2 , . . . ) 0 to a word w of evaluation (ρ1 , . . . , ρi−1 , ρi+1 , . . . , ρi+h , ρi , ρi+h+1 , . . . ). The oper(h) ation σi acts only on the subword w{i,...,i+h} and can thus be defined in generality from the special case i = 1. Let w be a word in 1, . . . , h + 1, and let (P(w), Q(w)) denote the pair of tableaux in RS-correspondence with w (see (2.14)). If w00 is the word obtained from w by first erasing all occurrences of the letter 1 and then decreasing the remaining letters by 1, then the shape of P(w00 ) differs from that of P(w) by a horizontal strip. Let T 0 be the tableau obtained from P(w00 ) by filling the horizontal strip with (h + 1)’s, and let w0 be the word that is in RS-correspondence with the pairs (h) of tableaux (P(w0 ), Q(w0 )) = (T 0 , Q(w)). We now define σ1 by setting (h)

σ1 (w) = w0 . (1)

(h)

(6.2) (h)

It can be shown that σi = σi , and thus σi generalizes σi . Note that σi to be a special case of an operation defined in [16].

happens

130

LAPOINTE, LASCOUX, and MORSE

The rectangular promotion operators are defined in a manner similar to the promotion operators. That is, on a tableau T of evaluation (λ1 , . . . , λm ), (h)

B(`h ) (T ) = σ1 · · · σm(h) R(`h ) T

(6.3)

generates a sum of tableaux with evaluation (`h , λ1 , . . . , λm ) by applying a rectangular analogue of Rr . This operator, R(`h ) , acts by adding to T a horizontal `-strip of the letter m + 1, a horizontal `-strip of m + 2, . . . , and a horizontal `-strip of m + h in all possible ways such that the tableaux are Yamanouchi in the added letters.∗ Since (1) σi = σi for h = 1, we recover the previously defined promotion operator B` : B(23 )

2 1 2

(3)

(3)

= σ1 σ2 R(23 ) 21 2  5 5 (3) (3) = σ1 σ2 + 2 4 4 1 2 3 3

=

3 3 2 2 5 1 1 4 5

+

4 3 3 2 2 1 1 5 5

5 4 5 2 4 1 2 3 3

+

4 3 2 1

5 3 2 1 5

5 4 2 1

+

+

5 4 3 2 3

5 3 3 2 2 5 1 1 4

+

+

5 4 5 2 3 4 1 2 3 5 4 3 3 2 2 1 1 5

+

+

5 4 3 5 2 4 1 2 3 5 4 3 2 1

5 3 2 1

+

5 4 3 2 1

5 4 3 2



(6.4)

In fact, it seems that the inverse of the B(`h ) action on an atom of level k is simply rectangular-katabolism. 20 (k) If τ is a translation of the letters in AT , we have CONJECTURE

(k)

(k)

K(`k−`+1 ) B(`k−`+1 ) AT = τ AT .

(6.5) (3)

The tableaux in (6.4) are sent, under K(23 ) , to 54 5 (a translation of the atom A212 ). Note that in general K(`h ) B(`h ) T is not necessarily equal to a translation of the letters in T . This conjecture supports the very important idea that any atom can be obtained from an irreducible atom simply by applying a sequence of rectangular promotion operators. 21 (k) (k) The operator B(`k−`+1 ) acts on any copy AT of Aλ by CONJECTURE

(k)

(k)

B(`k−`+1 ) AT = AT 0

(6.6)

for a tableau T 0 of shape λ ∪ (`k−`+1 ). ∗ This

is the multiplication involved in computing the Littlewood-Richardson coefficients in the product of a Schur function of the shape of T by a Schur function indexed by a rectangular partition.

TABLEAU ATOMS AND A POSITIVITY CONJECTURE (3)

For instance, by applying B(3) to A213 = B(3)

2 1 3

=

3 2 4 1 1 1

+

2 1 1 1 3 4

2 1 3

+

131 (3)

, we obtain a copy of A3,2,1 :

4 2 1 1 1 3

+

2 4 1 1 1 3

(3)

= A324111 .

(6.7)

Conjecture 21 not only reveals the importance of the set of irreducibles but also provides a convenient way to obtain copies using a simple transformation on tableaux. Given a tableau, the transformation LT is defined by replacing the shape(T )subtableau with T and then adjusting the remaining entries to start with s + 1, where s is the largest letter of T . LT satisfies several properties on the set of tableaux, denoted Hµ|(`h ) , with evaluation µ (not necessarily a partition) whose restriction to the h smallest letters gives exactly the subtableau of shape and evaluation (`h ). PROPERTY

22

Let T ⊆ Hµ|(`h ) be a set containing a unique element of minimal charge and whose

poset (T,
(k)

(k)

In particular, if we assume that AT 0 = B(`k−`+1 ) AT , then AT 0 ⊆ Hµ|(`h ) and, by (k)

Conjecture 10, the poset (AT 0 ,
COROLLARY 23 (k) (k) (k) If AT1 = B(`h ) AT2 is a copy of Aλ , then for each tableau T of shape (`h ), (k)

(k)

(6.8)

ALT T1 = LT AT1 (k)

is another copy of Aλ . (3)

In (6.7), we let 111 → 123 to obtain another copy of A3,2,1 : (3)

A546123 =

5 4 6 1 2 3

+

4 1 2 3 5 6

+

6 4 1 2 3 5

+

4 6 1 2 3 5

(6.9)

Proof of Property 22 Let T(`h ) be the tableau of shape and evaluation (`h ). Every element U ∈ T contains T(`h ) , and any letter in U/T(`h ) is larger than those in T(`h ) . Therefore the cyclage or

132

LAPOINTE, LASCOUX, and MORSE

cocyclage that links two elements U and U 0 of T does not involve the letters in T(`h ) , and we can thus change the content of this subtableau without affecting the cyclagecocyclage relations, as long as the new subtableau also contains the smallest letters. Hence the Hasse diagrams of the posets (T,
(k)

S(`k−`+1 ) Aλ = Aλ∪(`k−`+1 ) .

(6.10)

We have now seen that any atom can be understood as the application of rectangular promotion operators to an irreducible component. Our study is thus reduced to examining the irreducibles (atoms of level k that cannot be obtained by applying krectangular operators to a smaller atom). Interestingly, we can obtain the level k atom indexed by the irreducible partition of maximal degree,
λ M = (k − 1)1 , (k − 2)2 , . . . , 1k−1 , (6.11) by a recursive application of (k − 1)-rectangular promotion operators on the empty tableau. 25 The maximal irreducible atom of level k is an atom of level k − 1: CONJECTURE

(k)

(k−1)

Aλ M [X ; t] = Aλ M

[X ; t].

(6.12)

Furthermore, from Conjecture 21, this atom is simply (k)

(k−1)

Aλ M [X ; t] = Aλ M

[X ; t] = z(B(k−1) B((k−2)2 ) · · · B(1k−1 ) H0 ).

(6.13)

(3)

For example, the atom A2,1,1 [X ; t] is given by  (3) A2,1,1 [X ; t] = z(B(2) B(12 ) H0 ) = z

3 2 1 1

+

2 1 1 3



= S2,1,1 [X ; t] + t S3,1 [X ; t]. (6.14)

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

133

When t = 1, Vk = {Hλ [X ; t]}λ1 ≤k reduces to the polynomial ring Q[h 1 , . . . , h k ] = Vk (1). If Ik denotes the ideal generated by the k-rectangular Schur functions S(`k+1−` ) , we have the following proposition. PROPOSITION 26 The homogeneous functions indexed by k-irreducible partitions form a basis of the quotient ring Vk (1)/Ik .

Proof For a partition λ bounded by k, we set ( hλ h˜ λ = h˜ µ+(`k+1−` ) = S(`k+1−` ) h˜ µ

if λ is k-irreducible, if λ = µ ∪ (`k+1−` ).

(6.15)

These elements are indexed by k-bounded partitions and thus, if independent, span a space with the same dimension as Vk (1). In fact, the h˜ λ form a basis for Vk (1)
since S(`k+1−` ) = det h `−i+ j 1≤i, j≤k+1−` implies that h˜ λ ∈ Vk (1), and Sλ = h λ + P P ˜ µ>λ cµλ h µ gives h λ = h λ + µ>λ dµλ h µ , which implies that they are independent. First note that the h˜ λ span the quotient ring Vk (1)/Ik because they span Vk (1). Since, by definition, h˜ µ ≡ 0 in the quotient ring when µ is not k-irreducible, the h˜ λ

indexed by k-irreducible partitions form a basis for the quotient ring Vk (1)/Ik if they are independent in Vk (1)/Ik . Let S be the set of all k-irreducible partitions. If, in Vk (1)/Ik , we have X dλ h˜ λ = 0, (6.16) λ∈S

then, in Vk (1), we must have X λ∈S

dλ h˜ λ =

X

Ci S(i k+1−i )

(6.17)

i

P for some Ci ∈ Vk (1). Further, since Ci = µ ci,µ h˜ µ for some ci,µ , we have X X X ci,µ h˜ µ S(i k+1−i ) = ci,µ h˜ µ+(i k+1−i ) . (6.18) dλ h˜ λ = λ∈S

i,µ

i,µ

The basis elements appearing in the left-hand side of (6.18) are each indexed by kirreducible partitions, whereas those appearing in the right-hand side are indexed by non-k-irreducible partitions. Therefore dλ = 0 for all λ and, by (6.16), this proves that the h˜ λ indexed by k-irreducible partitions are independent in Vk (1)/Ik . We now have that the dimension of the quotient Vk (1)/Ik is k!. Since we assume that the atoms of level k form a basis for Vk , Corollary 6.10 implies that the k-irreducible

134

LAPOINTE, LASCOUX, and MORSE

atoms also form a basis of Vk (1)/Ik since the atoms generate Vk (1)/Ik , and the only possibly nonzero atoms in Vk (1)/Ik are the k! irreducible ones. COROLLARY 27 Assuming that the atoms of level k form a basis of Vk and that Conjecture 21 holds, the k-irreducible atoms form a basis of the quotient ring Vk (1)/Ik .

If we link all atoms that occur in the action of e1 on a given atom in Vk (1)/Ik , we obtain a poset illustrated in Figure 2 on page 109. The rank generating function of this poset was given in (1.26). This poset seems to have a remarkable symmetry property called flip-invariance. Definition 28 Given a k-irreducible partition of the form λ = (k − 1)n 1 , (k − 2)n 2 , . . . , 1n k−1




with n i ≤ i for all i,

(6.19)

the involution called flip f (k) is defined by (k)

f (k) Aλ = A where λf

(k)

(k) (k) , λf


= (k − 1)1−n 1 , (k − 2)2−n 2 , . . . , 1k−1−n k−1 .

For instance,

(5)

(6.20)

(6.21)

(5)

f (5) A4,3,2 = A3,2,2,1,1,1,1 .

CONJECTURE 29 The poset associated to the action of e1 on atoms in Vk (1)/Ik is flip-invariant. That (k) (k) is, if there is an arrow between two atoms Aµ and Aλ , then there will be an arrow (k) (k) between the two atoms A f (k) and A f (k) . µ

λ

Given the k! irreducible atoms, from which all other atoms are constructed using k-rectangular promotion operators, the complete decomposition of the standard tableaux into atoms can in principle be obtained. We give here the cases k = 2 and k = 3. 6.1. Cases k = 2 and k = 3 We start with k = 2. If Sn denotes the set of standard tableaux on n letters, then (B(2) + B(12 ) )Sn = Sn+2 ,

(6.22)

TABLEAU ATOMS AND A POSITIVITY CONJECTURE (2)

where B(2) = L and

(2) A1

=

1

135

B(2) and B(12 ) = B(12 ) . This recursion implies, for A0

1 2

= H0

, (B(2) + B(12 ) )` A(2)  = S2`+ ,

where  ∈ {0, 1}.

Expanding the left-hand side gives X B 3−v1 · · · B(v 3−vm ) A(2)  = S2`+ (v1

(v1 ,...,vm )

)

for vi ∈ {1, 2} ,

m

(6.23)

(6.24)

and each of the standard tableaux must occur in exactly one term of this sum. That is, each standard tableau must occur in exactly one family, denoted (2)

A(v1 ,...,vm ,) = B

3−v1

(v1

)

· · · B(v 3−vm ) A(2)  , m

vi ∈ {1, 2}.

(6.25)

We have thus decomposed the set of standard tableaux into these families, which are the atoms of level 2 by Conjecture 21. Furthermore, from Conjecture 6.5, given (2) a standard tableau, we can determine to which family A(v1 ,...,vm ,) belongs by first performing a (2)-katabolism (v1 = 2) if it contains the subword (12) and otherwise by performing a (1,1)-katabolism (v1 = 1). Repeating this procedure on the resulting tableau (until there is only one box left ( = 1) or no boxes left ( = 0)), we obtain the sequence (v1 , . . . , vm ) that we need. Now, from Conjecture 21, (2)

(2)

z(A(v1 ,...,vm ,) ) = t ∗ Aλ [X ; t],

(6.26)

3−vm where λ is the partition rearrangement of (v13−v1 , . . . , vm , ) and ∗ is a power of t. The symmetric function analogues of B(2) and B(12 ) are the vertex operators B(2) and B(12 ) (see Section 6.2). Therefore Conjecture 21 suggests that

B

3−v1

(v1

(2)

)

∗ · · · B(v 3−vm ) A(2)  [X ; t] = t Aλ [X ; t], m

 = 0, 1,

(6.27)

3−vm where λ is the partition rearrangement of (v13−v1 , . . . , vm , ) and ∗ is a power of t. This conjecture connects the atoms to the Macdonald polynomials since the creation operators that build the Macdonald polynomials recursively can be divided into the operators B(2) and B(12 ) (see [7], [21]). The positive expansion of Macdonald polynomials indexed by 2-bounded partitions (equivalently, partitions with `(λ) ≤ 2) into atoms of level 2 is thus, conjecturally, the one given in [7] and [21] (and in [18] since [19] proves the operators are related to the functions studied in [18]). In the case k = 3, we have the 8 irreducible atoms of 6 distinct shapes, (3)

(3)

A0 = H0 , (3) A21 (3)

=

2 1

A4312 =

A1 = (3) A312

, 4 3 1 2

+

3 1 2 4

,

=

1

3 1 2

(3)

,

A12 =

,

(3) A213

=

2 1 3

4 2 1 3

+

2 1 3 4

(3)

A4213 =

1 2

, , (6.28)

136

LAPOINTE, LASCOUX, and MORSE

from which we can build any atom of evaluation (1, . . . , 1) using the promotion operators B1

B3

4 1 2

2 3

,

B1

2 4

,

B1

B3 ,

B4 ,

B4 ,

2 1

2 1

3 1

,

B2

4 1 3

,

B3

5 1 2

3 4

,

,

(6.29)

B2

5 1 3

Here an operator indexed by a tableau T of shape R is LT B R followed by the reindexation of the letters not in T such that the resulting tableaux are standard. For instance, B1

3 4

2 1

=

5 2 1 3 4

+

(6.30)

2 1 3 4 5

Using (6.28) and (6.29), we consider the sets of tableau (3)

(3)

(6.31)

A(T1 ,...,Tm ,T ) = BT1 · · · BTm AT

for sequences (T1 , . . . , Tm , T ) which obey the following rules (read from right to left): 3 5 (1) and 21 53 can only follow a tableau that contains the subtableau 1 ; 1 2 2 (2) ; 1 2 4 and 1 3 4 can only follow a tableau that contains the subtableau 1 (3)

4 2 1

and

4 3 1

can only follow a tableau that contains the subtableau

1 2

.

We conjecture that there is a one-to-one correspondence between the sequences (T1 , . . . , Tm , T ) and the set of tableaux indexing all level 3 copy atoms with standard evaluation. Moreover, we can determine to which atom an arbitrary standard tableau belongs in view of Conjecture 6.5; katabolism is the inverse of rectangular promotion. That is, given a tableau U , we can determine which sequence (T1 , . . . , Tm , T ) can be extracted by katabolism from U . 6.2. Generalized Kostka polynomials Given a sequence of partitions S = (λ(1) , λ(2) , . . . , λ(m) ), the generalized Kostka polynomial HS [X ; t] found in [18] is a t-generalization of the product of Schur functions indexed by the partitions in S. (Different approaches to these polynomials include those in [8], [15].) More precisely, if we consider only its term of degree n = |λ(1) | + |λ(2) | + · · · + |λ(m) |, X HS [X ; t] = K λ;S (t) Sλ [X ], (6.32) λ`n

where, for the scalar product h , i on which the Schur functions are orthonormal,

 K λ;S (1) = Sλ [X ], Sλ(1) [X ] Sλ(2) [X ] · · · . (6.33)

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

137

If successively reading the entries of λ(1) , λ(2) , . . . produces a partition µ, S is said to be dominant. In this case, it has been conjectured (see [18]) that X HS [X ; t] = t charge(T ) Sshape(T ) [X ], (6.34) T ∈H S

where H S is the set of tableaux T of evaluation µ such that P S (T ) = T (see m 1 Section 3). Now if S = ((`k+1−` ), . . . , (`k+1−` )) is a dominant sequence of km 1 m 1 rectangles, then Conjecture 21 implies that for µ = (`k+1−` , . . . , `k+1−` ), m 1 B

k+1−`1 )

(`1

· · · B(`k+1−`m ) H0 = A(k) µ .

(6.35)

m

(k)

(k)

Moreover, by the definition of atoms, we have P S (Aµ ) = Aµ since µ→k = S. (k) Therefore H S = Aµ since both sets contain the same number of elements (the number of terms in the product of the Schur functions corresponding to shapes m 1 (`k+1−` ), . . . , (`k+1−` )). We thus have the following connection between atoms m 1 and the generalized Kostka polynomials. CONJECTURE 30 m 1 If S = ((`k+1−` ), . . . , (`k+1−` )) m 1

µ, then

k+1−`m 1 is such that (`k+1−` , . . . , `m ) is a partition 1

A(k) µ [X ; t] = HS [X ; t].

(6.36)

Further, it is shown in [19] that the generalized Kostka polynomials can be defined as HS [X ; t] = Bλ(1) Bλ(2) · · · Bλ(m) · 1,

(6.37)

where Bλ corresponds to Hλt in their notation. Given our formula (6.35), it is natural to assume that the vertex operators B(`k+1−` ) indexed by k-rectangular partitions are the operators that extend Conjecture 21 to the level of symmetric functions. CONJECTURE 31 Given a k-rectangular partition (`k+1−` ), we have (k)

(k)

B(`k+1−` ) Aλ [X ; t] = t c Aλ∪(`k+1−` ) [X ; t],

where c ∈ N.

(6.38)

7. The k-conjugation of a partition Here we introduce a generalization of partition conjugation, defined for partitions bounded by k. When k is large, our k-conjugation reduces to the usual conjugation. A skew diagram D is said to have hook lengths bounded by k if the hook length of any cell in D is not larger than k. For a positive integer m ≤ k, the k-multiplication m ×(k) D is the skew diagram D obtained by adding a first column of length m to D

138

LAPOINTE, LASCOUX, and MORSE

such that the number of parts of D is as small as possible while ensuring that its hook lengths are bounded by k. For example,

×(5)

(7.1)

=

Definition 32 Let λ = (λ1 , . . . , λn ) be a k-bounded partition, and let D be the skew diagram obtained by k-multiplying from right to left the entries of λ: D = λ1 ×(k) · · · ×(k) λn .

(7.2)

The k-conjugate of λ, denoted λωk , is the vector obtained by reading the number of boxes in each row of D. When k → ∞, λωk = λ0 since each k-multiplication step reduces to adding a column of length λi at the bottom row. PROPERTY 33 If λ is a k-bounded partition, then λωk is also a k-bounded partition.

Proof We know that λωk is k-bounded since D has hook lengths bounded by k. To see that λωk is a partition, assume that by induction the parts of D (2) = λ2 ×(k) · · ·×(k) λn form a partition µ. The skew diagram D = λ1 ×(k) D (2) is obtained by adding a column of length λ1 to D (2) starting at some row h. To see that D must also have parts of weakly decreasing size, it suffices to show that µh−1 > µh . Suppose µh−1 = µh , and consider the two possible cases (see Figure 6). Keep in mind that any column can be no longer than those to its left since λi ≤ λ j , ∀i > j. If row h − 1 lies directly below row h, then sliding the new column down to row h − 1 gives a skew diagram of length less than D with hook lengths at most k. Therefore our column would not have been added to row h. Now if row h − 1 lies below and to the right of row h, the column indicated by an arrow can be moved down without producing any hook lengths longer than k. Since D (2) = λ2 ×(k) · · · ×(k) λn , this is a contradiction. For example, we can compute (2, 2, 1, 1)ω4 = (3, 2, 1) by the following steps: ×(4)

×(4)

×(4)

=

×(4)

×(4)

=

×(4)

=

(7.3)

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

(1)

h−1

139

(2)

1111 0000 0000 1111 0000 1111

h−1

1111 0000 000 111 0000 1111 000 111

Figure 6

34 For a k-bounded partition λ, let D = λ1 ×(k) · · · ×(k) λn and let D be the skew diagram obtained by shifting any row in D to the left. If the number of columns of D is not more than the number of columns of D, then the hook lengths of D are not k-bounded. PROPERTY

Proof Assume that by induction D (2) = λ2 ×(k) · · · ×(k) λn , with rows of length µ, satisfies this property. The skew diagram D = λ1 ×(k) D (2) falls into one of the two generic cases illustrated in Figure 7. (1)

h

(2)

111 000 000 111

h

1111 0000 0000 1111

Figure 7

In the first case, since the column is added above row h, we know that λ1 + µh > k. Thus row h cannot be moved left or we would have a cell with hook length λ1 + µh > k. In the second case, row h cannot be moved left without violating the assumption that D (2) obeys the property. 35 The k-conjugation ωk is an involution on partitions bounded by k. That is, for λ with λ1 ≤ k, (λωk )ωk = λ. (7.4) THEOREM

Proof Let D = λ1 ×(k) · · ·×(k) λn . Property 34 implies that D is recovered by performing the

140

LAPOINTE, LASCOUX, and MORSE

k-multiplication of the entries of λωk in a conjugate way (adding rows to the leftmost position such that the hook lengths are never larger than k). Therefore, if λωk = µ, the conjugate of D is given by D 0 = µ1 ×(k) · · · ×(k) µm , and thus (D 0 )0 = D implies that µωk = (λωk )ωk = λ. Given the k-conjugation of a partition, it is natural to consider the relation between an atom indexed by λ and the atom indexed by λωk . In fact, our examples suggest that conjugating each tableau in an atom produces the tableaux in another atom. CONJECTURE 36 (k) (k) Let T be a standard tableau. For any copy AT of Aλ , (k)

(k)

(AT )t = AT 0

(7.5)

for some standard tableau T 0 of shape λωk . Since, at any level, there is at least one copy of each atom of a given degree in the set of standard tableaux, we have the following corollary. COROLLARY 37 In any atom of shape λ and level k, there is a unique element of maximal charge whose shape is the conjugate of λωk , (λωk )0 .

Furthermore, since a standard tableau T in n letters satisfies charge(T t ) = charge(T ), we have the following.

n
2 −

38 Let ω be the involution such that ωSλ [X ] = Sλ0 [X ]. Then, for some ∗ ∈ N, COROLLARY

(k)

(k)

ω Aλ [X ; t] = t ∗ Aλωk [X ; 1/t].

(7.6) (k)

Here we see that for large k, λωk = λ0 is consistent with the fact that Aλ [X ; t] = Sλ [X ] in this case. 8. Pieri rules Beautiful combinatorial algorithms are known for the Littlewood-Richardson coefficients that appear in a product of Schur functions: X ν Sλ Sµ = cλµ Sν . (8.1) ν

(k) Aλ [X ; t]

Recall that by Property 6 our atoms are simply the Schur functions Sλ when k is large. Therefore the expansion coefficients in a product of atoms are the

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

141

Littlewood-Richardson coefficients when k is large, and it is natural to examine the coefficients in a product of two atoms for general k. In fact, in the case t = 1, the coefficients in a product of two atoms do seem to generalize Littlewood-Richardson coefficients. CONJECTURE 39 (k) Let Aλ denote the

(k)

case t = 1 in Aλ [X ; t]. Then X (k) (k) ν ν (k) ν Aλ A(k) cλµ A(k) where 0 ⊆ cλµ ⊆ cλµ . µ = ν ,

(8.2)

ν

In particular, we know that ν (k) ν cλµ = cλµ

for k ≥ |µ|.

(8.3)

Identity (8.1) reduces to the Pieri rule when λ is a row (resp., column). Since an atom (k) Aλ reduces to h ` (resp., e` ) when λ is a row (column) of length ` ≤ k, our conjecture can be reduced to a k-generalization of the Pieri rule. COROLLARY 40 (k) (k) For certain sets of shapes E λ,` and E¯ λ,` , we have for ` ≤ k, (k)

h ` Aλ =

X (k)

µ∈E λ,`

A(k) µ

and

(k)

e` Aλ =

X

A(k) µ .

(8.4)

(k)

µ∈ E¯ λ,`

(k) (k) We conjecture that the sets E λ,` and E¯ λ,` can be defined in a manner analogous to the Pieri rule.

CONJECTURE 41 For any positive integer ` ≤ k,  (k) E λ,` = µ | µ/λ is a hori zontal `-stri p and µωk /λωk is a ver tical `-stri p ,  (k) E¯ λ,` = µ | µ/λ is a ver tical `-stri p and µωk /λωk is a hori zontal `-stri p . (8.5)

(4)

For example, to obtain the indices of the elements that occur in e2 A3,2,1 , we compute (3, 2, 1)ω4 = (2, 2, 1, 1) by Definition 32 and then add a horizontal 2-strip to (2,2,1,1) in all possible ways. This gives (2, 2, 2, 1, 1), (3, 2, 1, 1, 1), (3, 2, 2, 1), and (4, 2, 1, 1), of which all are 4-bounded. Our set then consists of all the 4-conjugates

142

LAPOINTE, LASCOUX, and MORSE

of these partitions that leave a vertical 2-strip when (3, 2, 1) is extracted from them. The corresponding 4-conjugates are (2, 2, 2, 1, 1)ω4 =

(3, 2, 1, 1, 1)ω4 =

,

(3, 2, 2, 1)ω4 =

(4, 2, 1, 1)ω4 =

,

,

(8.6)

and of these partitions, only the first three are such that a vertical 2-strip remains when (3, 2, 1) is extracted. Therefore (4)

(4)

(4)

(4)

e2 A3,2,1 = A3,3,2 + A3,2,2,1 + A3,2,1,1,1 ,

(8.7)

which is in fact correct. 9. Hook case (k) We are able to explicitly determine the functions Aλ [X ; t] in the case where λ is a hook partition and also to derive properties of atoms indexed by partitions slightly more general than hooks. These results rely on the following property of a row-shaped katabolism. PROPERTY 42 If T has shape λ = (m, 1r ) (a hook), then (

K(n) : T −→

T¯

if n ≤ m,

0

otherwise,

(9.1)

where T¯ is also hook-shaped. Proof Consider a tableau T of shape λ = (m, 1r ). If n > m, then T does not contain a row of length n and thus K(n) T = 0. Assume n ≤ m. Let U be the tableau of shape (1r ) obtained by deleting the bottom row of T . By the definition of katabolism, the action of K(n) on T amounts to row inserting a sequence of strictly decreasing letters (those of U ) into a sequence of weakly increasing letters (the last m − n letters in the bottom row of T ). The insertion algorithm implies (see [1]) that in this case no two elements may be added to the same row, and therefore we obtain a hook shape. This property leads to the hook content of any atom that is not indexed by a kgeneralized hook partition, that is, a partition of the form (k, . . . , k, ρ1 , ρ2 , . . .) for a hook shape (ρ1 , ρ2 , . . .).

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

143

PROPERTY 43 (k) If T is a tableau of shape λ, where λ is not a k-generalized hook, then AT does not contain any tableaux with a hook shape. COROLLARY 44 If λ is a partition that is not a k-generalized hook, then X (k) (k) Aλ [X ; t] = Sλ [X ] + vµλ (t) Sµ [X ],

(9.2)

µ>λ

(k)

where vµλ (t) = 0 for all hook partitions µ. Proof Let λ→k = (λ(1) , λ(2) , . . .). The condition on λ implies that λ2 is at least 2. If we first consider such partitions with λ1 6= k, then λ(1) cannot be a hook (λ1 6= k implies that the first partition in the k-split contains at least the first two parts of λ). But if λ(1) (k) is not a hook, then any hook-shaped tableau T in B(λ1 ) (A(λ2 ,λ3 ,... ) ) does not contain the shape λ(1) and is therefore sent to zero under Pλ→k . On the other hand, if λ1 = k, (k) then λ(1) = (k). Now any hook-shaped tableau T in B(λ1 ) (A(λ2 ,λ3 ,... ) ) is sent to a hook under the k-katabolism by Property 42. Our claim thus follows recursively on the remaining terms of the k-split of λ. If an atom is indexed by a k-generalized hook, we can determine it explicitly. PROPERTY 45 Let λ = (m, 1r ) be a k-irreducible hook partition. Then ( (r + 1) r · · · 2 1m if r + m ≤ k, (k) Aλ = m m (r + 1) r · · · 2 1 + r · · · 2 1 (r + 1) otherwise.

(9.3)

Note: Here an element (r + 1) r · · · 2 1m denotes the word (r + 1) r · · · 2 1 1 · · · 1. Proof Since r, m ≤ k − 1 in any k-irreducible partition λ = (m, 1r ), we have (1i )→k = (1i ) for 1 ≤ i ≤ r . Therefore, on a tableau T with i boxes, P(1i )→k T 6= 0 only for T of shape (1i ), and thus A1r = P(1r )→k B1 · · · P(12 )→k B1 P(1)→k B1 H0 = r r − 1 · · · 1.

(9.4)

Moreover, it develops that Bm A1r = (r + 1) r · · · 2 1m + r · · · 2 1m (r + 1). Now we have
A(m,1r ) = P(m,1r )→k (r + 1) r · · · 2 1m + r · · · 2 1m (r + 1) . (9.5)

144

LAPOINTE, LASCOUX, and MORSE

Since r + m − k ≤ k, the k-split of (m, 1r ) is ( ((m, 1r )) if r + m ≤ k, r →k (m, 1 ) = k−m r +m−k ((m, 1 ), (1 )) otherwise.

(9.6)

In either case, P(m,1r )→k ((r + 1) r · · · 2 1m ) = (r + 1) r · · · 2 1m , but since r · · · 2 1m (r + 1) never contains shape (m, 1r ), P(m,1r )→k (r · · · 2 1m (r + 1)) 6= 0 only in the second case. Now, by Conjecture 21, we use the given atoms of level k indexed by a k-irreducible hook shape to obtain more general cases including those indexed by a k-generalized hook shape. 46 Assume that Conjecture 21 holds. For a sequence of k-rectangles (R1 , R2 , . . . , R j ), let λ be the partition rearrangement of (R1 , R2 , . . . , R j , m, 1r ). Then COROLLARY


(k) Aλ [X ; t] ∝ z B R1 · · · B R j A(m,1r ) .

(9.7)

Acknowledgments. We give our deepest thanks to Adriano Garsia for all his time and effort helping us articulate our ideas. L. Lapointe would also like to thank Luc Vinet for his support and for helpful discussions. Our research depended on the use of ACE (see [20]). References [1]

[2]

[3] [4] [5] [6] [7]

W. FULTON, Young Tableaux: With Applications to Representation Theory and

Geometry, London Math. Soc. Stud. Texts 35, Cambridge Univ. Press, Cambridge, 1997. MR 99f:05119 113, 142 A. M. GARSIA and M. HAIMAN, A graded representation module for Macdonald’s polynomials, Proc. Natl. Acad. Sci. USA 90 (1993), 3607 – 3610. MR 94b:05206 103 A. M. GARSIA and C. PROCESI, On certain graded Sn -modules and the q-Kostka polynomials, Adv. Math. 94 (1992), 82 – 138. MR 93j:20030 105 M. HAIMAN, Hilbert schemes, polygraphs and the Macdonald positivity conjecture, J. Amer. Math Soc. 14 (2001), 941 – 1006. MR 2002c:14008 103 N. H. JING, Vertex operators and Hall-Littlewood symmetric functions, Adv. Math. 87 (1991), 226 – 248. MR 93c:17039 105 D. E. KNUTH, Permutations, matrices, and generalized Young tableaux, Pacific J. Math. 34 (1970), 709 – 727. MR 42:7535 112 L. LAPOINTE and J. MORSE, Tableaux statistics for two part Macdonald polynomials, preprint, arXiv:math.CO/9812001 135

TABLEAU ATOMS AND A POSITIVITY CONJECTURE

[8]

[9]

[10] [11]

[12] [13] [14] [15]

[16] [17] [18]

[19]

[20]

[21]

145

A. LASCOUX, B. LECLERC, and J.-Y. THIBON, Ribbon tableaux, Hall-Littlewood

functions, quantum affine algebras, and unipotent varieties, J. Math. Phys. 38 (1997), 1041 – 1068. MR 98c:05167 136 , “The plactic monoid” in Algebraic Combinatorics on Words by M. Lothaire, Encyclopedia Math. Appl. 90, Cambridge Univ. Press, Cambridge, 2002, 144 – 172. CMP 1 905 123 113 ¨ A. LASCOUX and M.-P. SCHUTZENBERGER , Sur une conjecture de H.O. Foulkes, C. R. Acad. Sci. Paris S´er. A-B 294 (1978), A323 – A324. MR 57:12672 104, 113 , “Le mono¨ıde plaxique” in Noncommutative Structures in Algebra and Geometric Combinatorics (Naples, 1978), Quad. “Ricerca Sci.” 109, Consiglio Nazionale delle Ricerche, Rome, 1981, 129 – 156. MR 83g:20016 113, 116, 121 I. G. MACDONALD, Symmetric Functions and Hall Polynomials, 2d ed., Oxford Math. Monogr., Oxford Univ. Press, New York, 1995. MR 96h:05207 104, 111 G. DE B. ROBINSON, On the representations of the symmetric group, Amer. J. Math. 60 (1938), 745 – 760. 112 C. SCHENSTED, Longest increasing and decreasing subsequences, Canad. J. Math. 13 (1961), 179 – 191. MR 22:12047 112 A. SCHILLING and S. O. WARNAAR, Inhomogeneous lattice paths, generalized Kostka polynomials and An−1 -supernomials, Comm. Math. Phys. 202 (1999), 359 – 401. MR 2000m:05234 136 M. SHIMOZONO, A cyclage poset structure for Littlewood-Richardson tableaux, European J. Combin. 22 (2001), 365 – 393. MR 2002g:05189 110, 129 , Multi-atoms and monotonicity of generalized Kostka polynomials, European J. Combin. 22 (2001), 395 – 414. MR 2002e:05148 110 M. SHIMOZONO and J. WEYMAN, Graded characters of modules supported in the closure of a nilpotent conjugacy class, European J. Combin. 21 (2000), 257 – 288. MR 2002b:05136 105, 110, 116, 135, 136 M. SHIMOZONO and M. ZABROCKI, Hall-Littlewood vertex operators and generalized Kostka polynomials, Adv. Math. 158 (2001), 66 – 85. MR 2002f:05158 110, 135, 136 S. VEIGNEAU, ACE: An algebraic combinatorics environment for the computer algebra system MAPLE, version 3.0, 1998, available from http://phalanstere.univ-mlv.fr/˜ace/ 143 M. A. ZABROCKI, A Macdonald vertex operator and standard tableaux statistics for the two-column (q,t)-Kostka coefficients, Electron. J. Combin. 5 (1998), R45. MR 2000a:05214 135

Lapointe Instituto de Matem´atica y F´ısica, Universidad de Talca, Casilla 747, Talca, Chile; [email protected]

146

LAPOINTE, LASCOUX, and MORSE

Lascoux Institut d’electronique et d’informatique Gaspard Monge, Universit´e de Marne-la-Vall´ee, 5 boulevard Descartes, Champs-sur-Marne, 77454 Marne-la-Vall´ee, CEDEX 2, France; [email protected] Morse Department of Mathematics, University of Miami, Coral Gables, Florida 33124, USA; [email protected]

MATROIDS, MOTIVES, AND A CONJECTURE OF KONTSEVICH PRAKASH BELKALE and PATRICK BROSNAN

Abstract We show that a certain class of varieties with origin in physics generates (additively) the Denef-Loeser ring of motives. In particular, this disproves a conjecture of M. Kontsevich on the number of points of these varieties over finite fields. Contents 0. Introduction . . . . . . . . . . . . . . . . 1. Preliminary results . . . . . . . . . . . . . 2. Determinantal schemes . . . . . . . . . . . 3. The matrix-tree theorem . . . . . . . . . . 4. Incidence schemes . . . . . . . . . . . . . 5. Extensions of bilinear forms . . . . . . . . 6. Reduction formulas . . . . . . . . . . . . 7. The module of a graph . . . . . . . . . . . 8. Matroid theory . . . . . . . . . . . . . . . 9. A counterexample to Kontsevich’s conjecture 10. Mn¨ev-Sturmfels universality . . . . . . . . 11. Forests . . . . . . . . . . . . . . . . . . . 12. Geometric motives . . . . . . . . . . . . . 13. Vector bundles and extensions . . . . . . . 14. Reduction formulas for GeoMot . . . . . . 15. Periods . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

147 154 156 160 161 163 164 166 169 170 172 174 175 179 183 185 187

0. Introduction In this paper, we show that a certain class of varieties with origin in the physics of Feynman amplitudes additively generates the Denef-Loeser ring of motives. This disproves a conjecture of Kontsevich on the number of points of these varieties over a DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 1, Received 31 October 2001. Revision received 3 December 2001. 2000 Mathematics Subject Classification. Primary 81Q30, 14G10; Secondary 05B35.

147

148

BELKALE and BROSNAN

finite field. It also enables us to investigate period integrals on these varieties and to show that the class of integrals obtained is quite general. 0.1. Kontsevich’s conjecture Let G be a finite graph with vertex set V = V (G), edge set E = E(G), and Betti numbers b0 (G) and b1 (G). Recall that a graph T is called a tree if b0 (T ) = 1 and b1 (T ) = 0. A subgraph T ⊂ G is called a spanning tree if T is a tree and V (T ) = V (G). For each edge e, let xe denote a formal variable. Consider the polynomial XY PG = xe , (0.1) T e6∈T

where the sum runs through all spanning trees of G. If G is not connected, PG = 0 because the sum is empty. Otherwise, PG is a homogeneous polynomial of degree b1 (G). The polynomial PG and other related polynomials appear in the analysis of electrical circuits. In the 19th century, these polynomials were studied by G. Kirchhoff, J. Maxwell, C. Borchardt, and J. Sylvester, and, for this reason, they are sometimes called Kirchhoff polynomials. An important property of Kirchhoff polynomials is that they have an expression in terms of determinants through the matrix-tree theorem (see [24]). In the combinatorics literature, Kirchhoff polynomials are also occasionally called unsignants because, while determinantal expressions usually involve minus signs, minus signs are conspicuously absent from PG . Kirchhoff polynomials also play a role in the evaluation of Feynman amplitudes. Let V (PG ) denote the scheme of zeros of PG over Z, a hypersurface in A E , and let YG denote its complement. Feynman amplitudes and their counterterms are then related to period integrals on the YG . (We refer the reader to [27, pp. 13 – 21], [2] for this relationship.) Motivated by computer calculations of the counterterms appearing in the renormalization of Feynman integrals (see [4], [14]), Kontsevich speculated that the periods of YG are multiple zeta values (MZVs). Under this assumption on the periods, it is natural to expect that the zeta functions associated to the YG are the zeta functions of motives of mixed Tate type (see [31]). Based on this hypothesis and the Weil conjectures, Kontsevich made a conjecture about the number of points of YG over a finite field (see [12]). To describe his conjecture, we first make a notational convention. For any scheme X of finite type over Z, let |X | denote the function q 7→ #X (Fq ). Thus |X | is a function from the set Q of prime powers to Z. Clearly, |X | determines the zeta function of X . We say that X is polynomially countable if |X | is a polynomial in Z[q].

MOTIVES AND A CONJECTURE OF KONTSEVICH

149

CONJECTURE 0.1 (Kontsevich) For all graphs G, YG is polynomially countable.

Since |V (PG )| + |YG | = q #E , this conjecture is equivalent to the conjecture that V (PG ) is polynomially countable. J. Stembridge [25] verified this conjecture for all graphs with fewer than 12 edges. For certain graphs it is relatively easy to see that the conjecture holds. For example, for G a cycle of length n, V (PG ) is isomorphic to An−1 , and thus |YG | = q n − q n−1 . We show, however, that Conjecture 0.1 is false. In fact, contrary to the extremely strong restrictions on the arithmetic nature of the schemes YG claimed by the conjecture, they are, from the standpoint of their zeta functions, the most general schemes possible. 0.2. Combinatorial motives and the main theorem To make this last statement precise, we introduce some notation. Let CMot+ denote the group generated by all functions of the form |X | for X a scheme of finite type over Z. We think of CMot+ as a coarse version of the ring of motives over Z. We discuss a finer ring of motives at the end of this introduction. As |X × Y | = |X ||Y |, CMot+ is a ring. And, as |A1 | = q, CMot+ is a Z[q]-module. We call CMot+ the ring of effective combinatorial motives. Let S be the saturated multiplicative system in Z[q] generated by the functions q n − q for n > 1. Set CMot = S−1 CMot+ . We remark that, since the functions in S are nonvanishing on Q , elements of CMot give everywhere-defined functions from Q to Q. We call CMot the ring of combinatorial motives. Let R = S−1 Z[q]. (We remark that R is a principal ideal domain; see [10].) Let CGraphs denote the R-module generated by all functions of the form |YG |. We can now state our main theorem. THEOREM 0.2 We have CGraphs = CMot .

Theorem 0.2 immediately implies that Conjecture 0.1 is false. For, if the conjecture were true, all functions of the form |X | would be in R . In particular, they would be rational functions. However, if we let X be the closed subscheme of A1Z defined by px = 0 for p a given prime, then |X |(q) = q if p|q and 1 otherwise. Thus |X | cannot be a rational function. Of course, other more interesting examples of X such that |X | is not rational exist. For example, let E/Z be an integral model of a smooth elliptic curve over Q. It is well known that |E| is not a polynomial, even if we restrict it to any “large” subset of Q . In particular, this gives a counterexample to the question in

150

BELKALE and BROSNAN

[23, p. 363] which asks if |YG | is always a quasi polynomial. 0.3. Stanley’s reformulation of Conjecture 0.1 The proof of Theorem 0.2 is based on R. Stanley’s reformulation of Kontsevich’s conjecture in terms of a polynomial Q G which is, roughly speaking, dual to PG . In [23], Stanley sets XY QG = xe , (0.2) T e∈T

where the sum again runs through all spanning trees. For G connected, Q G is homogeneous of degree #E(G) − b1 (G). Let X G = A E − V (Q G ). Stanley showed that Kontsevich’s conjecture is equivalent to the following analogous conjecture. 0.3 For all graphs G, |X G | ∈ Z[q]. CONJECTURE

In fact, we see in Proposition 1.2 that the R-submodule of CMot generated by the |X G | is exactly the same as the one generated by the |YG |. Thus, by Theorem 0.2, the |X G | also generate CMot . The schemes X G are, however, more tractable than the YG —particularly when the graph G is simple (i.e., has neither loops nor multiple edges) and has an apex. This is because, when G is simple, the polynomial Q G has an uncomplicated expression as a determinant via the matrix-tree theorem (see Sec. 3). This expression simplifies even further when G has an apex. (There is also an expression for PG as a determinant, but this expression seems unmanageable for our purposes.) We remind the reader that a vertex v is said to be an apex if there is an edge from v to every other vertex in G. Suppose that G is an arbitrary simple graph with vertex set V = {v1 , . . . , vn }. Then we form a graph G ∗ with an apex by simply adding a vertex v0 and connecting it by an edge to all other vertices. All graphs with an apex can be obtained through this process. Using the matrix-tree theorem, Stanley showed that, for any field K , X G∗ (K ) is isomorphic to the set of (n × n)-nondegenerate, symmetric matrices M satisfying the condition that Mi j = 0

if i 6= j and there is no edge from vi to v j .

(0.3)

Here i, j ∈ [1, n]. o be the scheme of all (n × n)-nondegenerate, symmetric matrices We then let Z G M satisfying condition 0.3 (see Sec. 3). Stanley’s observation essentially shows that o ∼ X ∗ . Thus the following conjecture, stated by Stembridge as [25, Conj. 7.1], ZG = G would follow from Conjecture 0.3.

MOTIVES AND A CONJECTURE OF KONTSEVICH

151

CONJECTURE 0.4 o is polynomially countable. For every simple graph G, Z G

Note that, while Conjectures 0.1 and 0.3 are trivial when G is disconnected, Conjecture 0.4 is not. This is related to the fact that the operation G 7→ G ∗ always produces a connected graph. However, we will see that Conjecture 0.4 is also false. For any subgraph H of G, let G − H be the graph obtained by removing the edges in H but leaving all vertices. Note that (G − H )∗ = G ∗ − H . If G is a simple graph with n vertices, then G is contained in the complete graph K n . We define the complement G o of G to be the graph K n − G. Note that (G o )∗ = (DG)o , where D is the operation of adding a disjoint vertex. It becomes convenient at this point to shift attention from G to its complement. o . When G has vertices {v , . . . , v } as above, Z is We therefore define Z G = Z G o n G 1 then the scheme of all (n × n)-matrices M satisfying the condition that Mi j = 0

if there is an edge from vi to v j .

(0.4)

We mention that many of the results obtained thus far on Conjecture 0.3 are most easily stated in terms of the |Z G |. For example, in [23, Th. 5.4], Stanley showed that Conjecture 0.3 holds when G = K n − K 1,s , where K 1,s is a star (one vertex connected by edges to s other vertices) and s ≤ n − 2. In the case of n = s + 2, G = 0 ∗ with o , and X = Z o = Z 0 = K s+1 − K 1,s . Thus 0 = K 1,s G K 1,s . It follows that Stanley’s 0 Theorem 5.4 is equivalent to the statement that |Z K 1,s | ∈ Z[q]. 0.4. Overview Let CGraphs∗ be the R-module generated by all functions of the form |Z G | for o | = |X o ∗ |, it is clear that CGraphs ⊂ G a simple graph. Since |Z G | = |Z G o (G ) ∗ CGraphs. Therefore the following theorem implies Theorem 0.2. THEOREM 0.5 We have CGraphs∗ = CMot .

The proof of Theorem 0.5 involves two steps. In the first step, we study certain incidence schemes A G (s, r, k). These schemes are defined so that, when K is a field, the K -points of A G (s, r, k) are the sets of pairs (Q, f ) with Q a symmetric bilinear form on K s of rank r and f a function from V (G) to K s whose span is of dimension k. The pair (Q, f ) is also subject to the incidence condition that
Q f (vi ), f (v j ) = 0 if there is an edge from vi to v j . (0.5)

152

BELKALE and BROSNAN

If G has n vertices, then |A G (n, n, n)| = |Z G ||GLn |. Since |GLn | ∈ R , this implies that |A G (n, n, n)| ∈ CGraphs∗ . Moreover, there are important relations between the A G (s, r, k) for varying s, r , and k, and between the A G (s, r, k) for varying G. By exploiting these relations, we see that the R-module generated by the |A G (s, r, k)| is exactly CGraphs∗ . This fact allows us to shift our focus from the symmetric form Q to the function S f . In particular, for each s we consider the scheme JG (s) = k A G (s, s, k). Again, it turns out that the R-module generated by the JG (s) is exactly CGraphs∗ . Moreover, the JG (s) are quite manageable schemes because the dimension of the span of f is allowed to vary. The second step in our proof of Theorem 0.5 involves comparing the JG (s) to the representation spaces of matroids. For any matroid M, we define a scheme X (M, s). For K a field, X (M, s)(K ) is the set of all possible representations of M in K s . We then let CMatroids denote the R-module generated by all functions |X (M, s)|. As we see in Section 10, it follows from Mn¨ev’s universality theorem in [19] that CMatroids = CMot . On the other hand, we prove that, for each matroid M, there is a finite set of graphs {G i } and rational functions ai ∈ R such that X |X (M, s)| = ai |JG i (s)|. (0.6) This equation proves that CMatroids ⊂ CGraphs∗ , and thus it proves Theorem 0.5. Moreover, as we see, (0.6) can be used even without Mn¨ev universality to produce a contradiction to Conjecture 0.4. This is because there are matroids M, for example, the Fano matroid, which are representable only over fields of characteristic 2. Thus, for such matroids, |X (M, r )| (with r equal to the rank of M) could not possibly be a rational function as Conjecture 0.4 and (0.6) would demand. As Conjecture 0.1 implies Conjecture 0.4, this shows that Conjecture 0.1 is false. 0.5. Forest complements A considerable amount of work has been done to find examples of graphs for which Conjecture 0.1 (resp., Conj. 0.3, Conj. 0.4) holds and to compute the functions |YG | o |) explicitly (see [5], [23], [25], [30]). It remains an interesting ques(resp., |X G |, |Z G tion to determine the largest classes of graphs for which these conjectures are valid. The class of graphs for which Conjecture 0.3 holds is already known to include several important examples. Stanley showed that X K n −K m is polynomially countable. F. Chung and C. Yang then computed the polynomial |X K n −K m | explicitly (see [5]). Yang showed that X G is polynomially countable when G is an outplanar graph. And, as mentioned above, a consequence of [23, Th. 5.4] is that Z K 1,s is polynomially countable. Recall that a forest is a graph with no cycles. In Section 11, we show that Z F is polynomially countable whenever F is a forest. This generalizes Stanley’s Theorem

MOTIVES AND A CONJECTURE OF KONTSEVICH

153

5.4 and implies that Conjecture 0.4 holds for forest complements. The result is essentially a consequence of the manageability of the schemes J F (s), which allows us to compute |J F (s)| inductively in terms of the |J F 0 (s)| for smaller forests F 0 . 0.6. Geometric motives We have written the majority of this paper in terms of combinatorial motives because they suffice for the proof that Kontsevich’s conjecture is false. However, the reader who is familiar with the Kontsevich-Denef-Loeser theory of motivic integration will see that the statements in the paper are valid in a finer setting once the combinatorial process of counting points is replaced with the algebraic process of stratifying a scheme into disjoint subschemes. (This is, in fact, the process used by Stembridge’s Maple program in [25] to verify Kontsevich’s conjecture for graphs with less than 12 edges.) Following J. Denef and F. Loeser [7] and A. Craw [6], we define the ring of motives as follows. Write GeoMot+ for the abelian group generated by the symbols [X ] for X a scheme of finite type over Z modulo the following relations: (a) [X ] = [Y ] if X ∼ = Y, (b) [X ] = [X − V ] + [V ] if V is closed in X . The group GeoMot+ becomes a ring once we check that it is consistent to define [X ][Y ] = [X × Y ]. This ring, which we call the ring of effective geometric motives over Z, has [Spec Z] as its unit. In Section 12, we define the ring of effective geometric motives over an arbitrary base, and we give several results concerning GeoMot+ which we hope are of independent interest. There is an obvious surjection ev : GeoMot+  CMot+ given by sending [X ] to |X |. Following tradition, we write L for [A1 ] and note that ev(L) = q. (L is known as the Tate motive.) Clearly, the evaluation map restricted to Z[L] is an isomorphism onto its image, which is Z[q]. We therefore write S for the saturated multiplicative subset of Z[L] generated by Ln − L for n > 1 and R for the localization S−1 Z[L]. That is, in this paper, we use the symbols S and R in the context of CMot and in the context of GeoMot . We hope that this slight abuse of notation does not lead to confusion. We write GeoMot = S−1 GeoMot+ . This ring, which we call the ring of geometric motives, essentially appears in the work of Denef, Loeser, and Craw on motivic integration (see, e.g., [7, Cor. 6.3.4] or [6, Cor. 1.18]). When n > 1, Ln − L is invertible in the completion of the ring of motives where Kontsevich’s motivic measure takes its values. We now state the main results of our paper in the context of geometric motives.

154

BELKALE and BROSNAN

THEOREM 0.6 Let Graphs be the sub-R-module of GeoMot generated by the [YG ], and let Graphs∗ be the sub-R-module of GeoMot generated by the [Z G ], where G runs over all simple graphs. Let Forests denote the sub-R-module of GeoMot generated by the [J F (s)] for all integers s and all forests F. We have (a) Graphs∗ = Graphs = GeoMot , (b) Forests = R .

The advantage of having these statements in the context of geometric motives is that we can use them to investigate geometric invariants of the graph schemes. In particular, we can study the original motivation for Kontsevich’s conjecture, namely, the periods of the varieties defined by Kirchhoff polynomials. In Section 15, using the “stringy” E-polynomials of motivic integration (see [6]), we show that there is a graph G and two integers p 6= q such that the Hodge-Deligne number h p,q (YG ) is nonzero. If one accepts a recent conjecture of Kontsevich and D. Zagier [13] concerning the nature of periods, this implies that the periods of the YG are, in fact, not always multiple zeta values. The verification of Theorem 0.6 is left to the end of the paper, where we point out the modifications needed to turn counting arguments into algebro-geometric ones. Much of this is routine and left to the reader. However, the burden of working with motives over Spec Z is daunting enough that stating everything the first time around in terms of geometric motives would obscure the logic of the arguments significantly. 1. Preliminary results In this section, we carry out two minor adjustments to two theorems of Stanley. 1.1. The module of all graphs The first adjustment is an amplification of [23, Prop. 2.1]. It concerns the relation between the schemes YG and the schemes X G . Here we work in GeoMot+ . 1.1 The subgroup of GeoMot+ generated by the [X G ] is equal to the subgroup generated by the [YG ]. PROPOSITION

We remark that the proof of Proposition 1.1 is completely contained in Stanley’s proof of [23, Prop. 2.1]. However, for the convenience of the reader, we translate Stanley’s proof into our own setting.

MOTIVES AND A CONJECTURE OF KONTSEVICH

155

Proof Let S be a subset of E = E(G). Let A S be the image of the obvious inclusion i S : S = i S (G#S ). Note that, as S varies over all subsets of E, the A#S → A E . Let Gm m S stratify A E . subschemes Gm E−S For any subscheme X ⊂ A E , let X S = X ∩ A E−S (resp., X + S = X ∩ Gm ). Thus X S is the intersection of X with the hyperplanes defined by the equations xe for e ∈ S. Note that X ∅ = X , and, as S varies over the subsets of E, the subschemes X + S stratify X . We therefore have X [X S ] = [X T+ ] (1.1) T ⊃S

and, by the inclusion-exclusion principle, X [X + (−1)#(T −S) [X T ]. S]=

(1.2)

T ⊃S + ∼ + By inspecting the Q G , it is easy to see that X G,S ∼ . = X G−S and X G,S = X G−S + ∼ + ∼ Dually, if S is a forest, YG,S = YG/S (resp., YG,S = YG/S ), where G/S is the graph obtained by contracting each component of S to a point. On the other hand, if S is not a forest, it is easy to see that YG,S is empty. Q Now, as Stanley notes, Q G (x) = PG (1/x) e∈E xe . Thus + ∼ + X G,∅ = YG,∅

(1.3)

+ through the map x 7→ 1/x, that is, by sending the point x = (xe )e∈E of X G,∅ to the + point in YG,∅ with coordinates (1/xe )e∈E . Putting our equations together, we obtain the following: X X [YG ] = (−1)#T [X (G/S)−T ], (1.4) S⊂E T ⊂G/S b1 (S)=0

[X G ] =

X X

(−1)#T [Y(G−S)/T ].

(1.5)

S⊂E T ⊂E−S b1 (T )=0

Together, these two equations, the first of which appears (in a different notation) as [25, Prop. 4.1], prove the proposition. Proposition 1.1 implies the following proposition as a corollary. PROPOSITION 1.2 Graphs is equal to the R-submodule of GeoMot spanned by the [X G ]. CGraphs is equal to the R-submodule of CMot spanned by the |X G |.

156

BELKALE and BROSNAN

We remark that, as [YG ] = Ln −Ln−1 for G a cycle of length n, R is itself a submodule of Graphs. 1.2. An observation on polynomial countability Our second adjustment to Stanley’s results is to [23, Prop. 2.2]. This proposition, which Stanley deduces from the Weil conjectures, essentially states that, if X is a scheme of finite type over Z, then the knowledge that |X | ∈ Q[q] implies that, in fact, |X | ∈ Z[q]. In Section 11, we require a result that is analogous to Stanley’s Proposition 2.2 but easier to prove. While the result is not strictly weaker than Stanley’s proposition, it does not require the Weil conjectures. Rather, it is a consequence of the Euclidean algorithm. 1.3 If f ∈ R and f (q) ∈ Z for all q ∈ Q , then f ∈ Z[q]. PROPOSITION

We use Proposition 1.3 in the case of f = |X | for X a scheme of finite type over Z. Proof Write f = a/s with a ∈ Z[q] and s ∈ S. Since s is monic, we can write f = d + r/s with d, r ∈ Z[q] and deg(r ) < deg(s). But this implies that r (q)/s(q) ∈ Z for all q, which implies that r = 0. Thus f = d. 2. Determinantal schemes In this section, we collect certain basic properties of determinantal schemes which are necessary for the definition of the incidence schemes A G (s, r, k). We first describe the general theory of determinantal schemes in functorial language and then restrict to the specific cases of determinantal schemes over Z that are the focus of the paper. These results are necessary for Theorem 0.6, but they are not strictly necessary for the proof that Conjecture 0.1 is false. The reader who is interested only in Kontsevich’s conjecture may therefore skim this section until the end, where we state formulas for the motives of four important types of determinantal schemes. 2.1. Degeneracy loci Let S be a scheme, and let E and F be two locally free O S -modules of ranks e and f , respectively. Let φ : E → F be a morphism. The r th degeneracy locus Dr (φ) of φ is the closed subset consisting of all points s ∈ S such that φ ⊗ k(s) has rank less than or equal to r . We put a structure of a closed subscheme on Dr (φ) by writing it as the

MOTIVES AND A CONJECTURE OF KONTSEVICH

157

scheme of zeros of the morphism r^ +1

φ:

r^ +1

E→

r^ +1

F.

(2.1)

Equivalently, Dr (φ) is the closed subscheme of S corresponding to the ideal generated by the ((r + 1) × (r + 1))-minors of φ. Note that the subschemes Z r (φ) = Dr (φ) − Dr −1 (φ) partition S into a disjoint union of locally closed subschemes (see [9, Chap. 14] for more details). 2.2. Determinantal schemes With S, E, and F as above, we write HomO S (E, F) for the abelian group of all homomorphisms from E to F. The scheme of homomorphisms Hom(E, F) is then an abelian group scheme over S representing the functor T

HomOT (E T , FT ).

(2.2)

Write π : Hom(E, F) → S for the structure map. Hom(E, F) is then equipped with a universal map φ : π ∗ E → π ∗ F. The fact that Hom(E, F) represents the homomorphism functor can be expressed by saying that, for any S-scheme T and any map ψ : E T → FT , there is a unique map T → Hom(E, F) such that ψ ∼ = φT . We write Hom≤r (E, F) for the degeneracy locus Dr (φ), and we write Homr (E, F) for the locally closed subscheme Z r (φ). We call both types of schemes determinantal schemes. The Homr (E, F) are important in this paper as they stratify Hom(E, F) into a disjoint union of locally closed schemes. They also represent a natural functor, and the functorial description is useful as a language for describing other schemes in terms of the Homr (E, F) and for defining maps from the Homr (E, F) to other schemes. Let us say that the rank of a morphism ψ : E → F is r if the cokernel of ψ is a locally free sheaf on S of rank f − r . PROPOSITION 2.1 Let S be a Noetherian scheme; then Homr (E, F) represents the functor  T ψ : E T → FT rk(ψ) = r .

(2.3)

Proof Suppose that we are given a scheme T and a morphism ψ : E T → FT . By the universal property of Hom(E, F), this information gives us a unique morphism σ : T → S such that ψ = φT , where φ is the universal morphism. We need to show that this morphism σ factors through Homr (E, F) if and only if rk(ψ) = r . Now σ factors through Homr (E, F) if and only if the pullback of the sheaf of ideals defining Hom≤r (E, F) is zero on T and Dr −1 (ψ) = ∅. This is the case if

158

BELKALE and BROSNAN

and only if the ((r + 1) × (r + 1))-minors of ψ are zero while some (r × r )-minor is invertible in OT . These two conditions are local on T . We conclude the proof by recalling the following standard result. LEMMA 2.2 Suppose that A is a Noetherian local ring and ψ : Ae → A f is a morphism. Then the following are equivalent: (a) coker ψ is free of rank f − r ; (b) every ((r + 1) × (r + 1))-minor of ψ is zero, but some (r × r )-minor of ψ is invertible.

Proof See [8, Prop. 20.8, p. 495]. This concludes the proof Proposition 2.1. It is useful to have an explicit local description of our determinantal schemes in terms e f of coordinates and ideals. Let {yi j }i=1 j=1 be a set of formal variables, and consider each yi j as an entry in an (e × f )-matrix. Let A[y] be the polynomial ring in all variables yi j . For each k, let m lk ∈ Z[yi j ] be a complete list of the (k × k)-minors, and f let Ik be the ideal generated by the m lk . In this notation, Homr (O Se , O S ) is the locally f closed subscheme of Hom(O Se , O S ) given by the union of the affine schemes [ Spec(A[y]/Ir +1 )(m ri ) . (2.4) i f

It follows that the set of points associated to Homr (O Se , O S ) is simply T r i V (m i ).

T

i

V (m ri +1 ) −

2.2.1. Maps to the Grassmanian Write Gr(r, E) for the Grassmanian of r planes in E. This is defined to be the scheme representing the functor  T K ⊂ E E/K is locally free of rank e − r . (2.5) Homr (E, F) is equipped with two maps to Grassmanians. We have a map p : Homr (E, F) → Gr(r, F) given by sending a map φ to its image. And we have a map q : Homr (E, F) → Gr(e − r, F) given by sending φ to its kernel. 2.2.2. Function spaces When V is a finite set, we write Fun(V, E) for Hom(O SV , E) (resp., Funr (V, E) for Homr (O SV , E)).

MOTIVES AND A CONJECTURE OF KONTSEVICH

159

2.2.3. Symmetric bilinear forms Let E ∨ denote the dual of E. There is a natural transpose automorphism t : Hom(E, E ∨ ) → Hom(E, E ∨ ),

(2.6)

and we define Sym E to be the subscheme fixed by t. We then write Symr E (resp., Sym≤r E) for the scheme-theoretic intersection of Sym E with Homr (E, E ∨ ) (resp., Hom≤r (E, E ∨ )). 2.3. A specific case f We are primarily interested in the case of S = Spec Z, E = O Se , and F = O S . In this case, Hom(E, F) is Spec Z[y]. Hom≤r (E, F) is the closed subscheme in Hom(E, F) defined by the ((r + 1) × (r + 1))-minors. And Homr (E, F) is the Zariski open subset of Hom≤r (E, F) defined by requiring at least one (r × r )-minor to be invertible. These equalities can be used without reference to the preceding general theory (Prop. 2.1) to define the Homr (E, F). It follows directly that for any field K , Homr (E, F)(K ) is the set of maps from K e to K f of rank r . Similarly, when E = O Se with S = Spec Z, Sym E can be viewed as the closed subscheme of Z[y] defined by the equations yi j = y ji . Sym≤r E is then the closed subscheme of Sym E defined by the ((r + 1) × (r + 1))-minors. And Symr E is the Zariski open subset of Sym≤r E defined by requiring at least one (r × r )-minor to be invertible. The K -points of Symr E are the bilinear forms on K e of rank r . 2.4. Polynomial countability f When E = O Se , F = O S , and S = Spec Z, we write Homr (e, f ) for Homr (E, F), GLe for Home (e, e), Gr(r, e) for Gr(r, E), and Symre for Symr E. We now list a few results concerning the polynomial countability of the schemes just discussed: |GLn | = (q n − 1)(q n − q) · · · (q n − q n−1 ), |GLb | , |Gr(a, b)| = a(b−a) q |GLa ||GLb−a | |Homr (e, f )| = |Gr(r, e)||Gr(r, f )||GLr |.

(2.7) (2.8) (2.9)

The first two of the above equalities are well known, and the last is easy. Note that each of the functions given is a polynomial lying in the multiplicative set S. The following formula of J. MacWilliams [17] is more difficult: Q Q2s−1 n−i q 2i  s − 1), 0 ≤ r = 2s ≤ n, i=1 q 2i −1 · i=0 (q n |Symr | = Qs (2.10) Q2s q 2i n−i  − 1), 0 ≤ r = 2s + 1 ≤ n. i=1 q 2i −1 · i=0 (q

160

BELKALE and BROSNAN

Note again that |Symrn | ∈ S. All four of these equalities remain valid in the ring GeoMot once q is replaced with L. 3. The matrix-tree theorem Stanley’s positive results mentioned in the introduction were mainly consequences of the matrix-tree theorem of Kirchhoff, Borchardt, and Sylvester, which gives an expression of the polynomial Q G as the determinant of a symmetric matrix. As this theorem is also basic to our results, we describe it in this section after fixing some useful notation. 3.1. Notation When G is a simple graph, an assumption we make for the remainder of this paper, E can be considered a subset of Sym2 V . For v, w ∈ V , we write evw for the set {v, w}. Thus the statement evw ∈ E means that there is an edge in G connecting v to w. It is convenient to pick an ordering V = {v1 , . . . , vn G } of the vertices V , where n G = #V (G). We write n for n G when there is only one graph under consideration. Set ei j = evi v j . We write xi j for the variable xei j when ei j ∈ E, and we extend this notation by setting xi j = 0 when ei j 6∈ E. 3.2. The Laplacian Let L = L i j be the (n × n)-matrix defined by (P n Li j =

k=1 x ik

−xi j

if i = j, if i 6= j.

Let L 0 be L with the first row and the first column removed. L is called the generic Laplacian matrix of G and L 0 the reduced generic Laplacian. The following theorem can be found in the work of A. Cayley, Kirchhoff, Maxwell, and Sylvester. For a proof, see [24]. THEOREM 3.1 (The matrix-tree theorem) We have Q G = det L 0 . 0 be the scheme of all (n × n)-symmetric, nondeNow, as in the introduction, let Z G generate bilinear forms Mi j such that Mi j = 0 whenever i 6= j and ei j 6∈ E. In the 0 is simply the closed subscheme of Symn defined by the notation of Section 2, Z G n equations yi j = 0 for all i 6= j with ei j 6∈ E. Our use of Theorem 3.1 is based on the following important consequence, recognized by Stanley.

MOTIVES AND A CONJECTURE OF KONTSEVICH

161

THEOREM 3.2 o. We have X G ∗ ∼ = ZG

Proof Let Z[x] be the ring generated by the variables xi j for 0 ≤ i < j ≤ n. Let I be the ideal generated by the variables xi j for all pairs i < j with ei j 6∈ E. Then X G ∗ = Spec A with A = (Z[x]/I ) Q G . On the other hand, let Z[y] be the ring generated by all yi j for i, j ∈ {1, . . . , n}, and let J be the ideal generated by all expressions of the form yi j − y ji for i 6= j and yi j for i 6= j and ei j 6∈ E. Then, letting D be the determinant of the matrix of yi j ’s, 0 = Spec B with B = (Z[y]/J ) . ZG D Let p : Z[y] → Z[x] be the map P P    k
Let q : Z[x] → Z[y] be the map (P

k

y jk , i = 0,

−yi j ,

i > 0.

(3.2)

It is easy to verify that p(I ) ⊂ J , that q(J ) ⊂ I , and that p and q give inverse isomorphisms between the rings Z[x]/I and Z[y]/J . It then follows from the matrixtree theorem that p(Q G ) = D. Thus p and q give inverse isomorphisms between the rings A and B. As mentioned in the introduction, it is convenient to shift our attention from the simple o . Thus Z is the subscheme graph G to its complement. We therefore set Z G = Z G o G of Symnn defined by the equations yi j = 0 for every pair i, j with ei j ∈ E, and Z G = X (G o )∗ = X (DG)o , where D is the operation of adding a disjoint vertex. Example 3.3 Let G be a graph with n vertices and no edges. Then Z G ∼ = Symnn . This is recognized in [23]. By equation (2.10), it follows that |Z G | ∈ Z[q]. In fact, |Z G | ∈ S, and this shows that R ⊂ CGraphs∗ . 4. Incidence schemes We now introduce the incidence schemes mentioned in the introduction. At first, we work in full generality over a base scheme S. But our main interest is the case of S = Spec Z.

162

BELKALE and BROSNAN

Definition 4.1 Let W be a locally free O S -module, and let G be a graph. We write A G (W ) for the closed subscheme of Sym W × S Fun(V, W ) representing the functor 
T (Q, f ) ∈ Sym WT × Fun(V, WT ) Q f (v), f (w) = 0 if evw ∈ E(G) . (4.1) If r and k are integers, we write A G (W, r, k) for
A G (W ) ∩ Symr (W ) × S Funk (V, W ) . That is, A G (W, r, k)(T ) consists of pairs (Q, f ) ∈ A G (W ) such that Q has rank r and f has rank k. When S = Spec Z and W = O Ss , we write A G (s) for A G (W ) and A G (s, r, k) for A G (W, r, k). The A G (W, r, k) form a stratification of A G (W ) by locally closed subschemes. Note that A G (s, r, k) is empty unless 0 ≤ k < n and 0 ≤ r, k ≤ s. Also, note that A G (s, r, 0) = Symrs and A G (W, 0, k) = Funk (V, W ). Thus |A G (s, r, 0)| and |A G (s, 0, k)| are both in Z[q]. Now assume that V (G) = {v1 , . . . , vn } as in Section 3.1. Recall from the introduction that CGraphs∗ is the R-submodule of CMot spanned by the functions |Z G |. 4.2 We have A G (n, n, n) ∼ = Z G × GLn . CGraphs∗ is exactly equal to the R-module generated by the functions |A G (n, n, n)|.

PROPOSITION

(a) (b)

Proof We first remark that (b) follows directly from (a) and the fact that |GLn | ∈ S. To prove (a), we let W = O Sn with S = Spec Z. Then Funn (V, W ) = GLn . The map (Q, f ) 7→ ( f t Q f, f ) then identifies A G (n, n, n) with Z G × GLn . (Here f t denotes the transpose of f .) Remark 4.3 Let Z G (r ) be the scheme consisting of all (n × n)-symmetric bilinear forms of rank r such that Mi j = 0 whenever ei j ∈ E. These schemes have been studied implicitly in [5] and [23]. In Stanley’s notation, |Z G o (r )| = h(G, r ), and Chung and Yang call a graph G strongly admissible if h(G, r ) is a polynomial. An easy modification of the proof of Proposition 4.2 shows that A G (n, r, n) ∼ = Z G (r ) × GLn .

MOTIVES AND A CONJECTURE OF KONTSEVICH

163

5. Extensions of bilinear forms In this section, we review a result of MacWilliams [17] counting the number of ways to extend a bilinear form of rank r1 to a bilinear form of rank r2 . This count is important in Section 6 for finding relations among the A G (s, r, k). Let Q be a fixed bilinear form on Fqd1 with rank r1 . Let C Q (d2 , r2 , d1 , r1 ) be the number of ways to extend Q to a form on Fqd2 of rank r2 . The following result is [17, Lem. 4]. THEOREM

5.1

We have   q r1 ,    q r1 +1 − q r1 , C Q (d1 + 1, r2 , d1 , r1 ) =  q d1 +1 − q r1 +1 ,     0

r2 = r1 , r2 = r1 + 1, r2 = r1 + 2, otherwise.

Note that C Q (d1 + 1, r2 , d1 , r1 ) depends only on d1 , r2 , and r1 . By induction on d2 − d1 , we can show that C Q (d2 , r2 , d1 , r1 ) depends only on the integer parameters d2 , r2 , d1 , and r1 . Thus we simply write C(d2 , r2 , d1 , r1 ) for this number. We can also see by induction that the following recursion is satisfied: C(d2 , r2 , d1 , r1 ) =

2 X

C(d2 , r2 , d1 + 1, r1 + j)C(d1 + 1, r1 + j, d1 , r1 ).

(5.1)

j=0

5.2 C(d2 , r2 , d1 , r1 ) is a polynomial in q. C(d2 , r2 , d1 , r1 ) 6= 0 if and only if d2 ≥ r2 , d1 ≥ r1 , and 0 ≤ r1 ≤ r2 ≤ r1 + 2(d2 − d1 ).

COROLLARY

(a) (b)

Proof Part (a) follows directly from the recursion formula (5.1). The necessity of the first two inequalities of (b) is obvious for dimension reasons. (The rank of a bilinear form cannot be greater than the dimension of the ambient space.) The necessity of the third inequality follows from formula (5.1) by induction. We prove the sufficiency of the inequalities in (b) by induction on i = d2 − d1 using formula (5.1). We do not actually need this for the rest of the paper, so the reader may safely skip the proof. For i = 0 sufficiency is obvious. For i = 1 sufficiency results from the fact that C(d1 + 1, r2 , d1 , r1 ) 6= 0 if and only if r2 ∈ [r1 , r1 + 2] when d1 6= r1 and if and only if r2 ∈ [r1 , r1 + 1] when d1 = r1 .

164

BELKALE and BROSNAN

Now suppose that sufficiency is known for d2 − d1 < i, and assume that (d2 , r2 , d1 , r1 ) satisfies the conditions in (b) with d2 = d1 + i and r2 = r1 + k. By formula (5.1), C(d2 , r2 , d1 , r1 ) 6= 0 if there is a j such that both (1) C(d1 + i, r1 + k, d1 + 1, r1 + j) 6= 0 and (2) C(d1 + 1, r1 + j, d1 , r1 ) 6= 0. One computes that (2) is satisfied whenever j ≤ d1 − r1 + 1. Using the induction hypothesis, we see that (1) is satisfied for k − 2i + 2 ≤ j ≤ min(k, d1 − r1 + 1). So we need only show that k −2i +2 ≤ min(k, d1 −r1 +1). The inequality k −2i +2 ≤ k says only that i ≥ 1, which we are of course assuming. And k−2i +2 ≤ d1 −r1 +1 if and only if (d2 −d1 )+(d2 −r2 ) ≥ 1, which then follows from the fact that d2 ≥ r2 . 6. Reduction formulas In this section, we give three formulas (Props. 6.1, 6.2; Cor. 6.3) which allow us to reduce questions about A G (s, r, k) for given s, r , or k to questions where s, r , or k is smaller. We also give a formula (Prop. 6.4) that allows us to connect the A G (s, r, k) to the A DG (s, r, k), where DG, as in the introduction, is the graph obtained from G by adding a disjoint vertex. In the proof of Propositions 6.1 and 6.2, we pick a base field Fq at the beginning, and then, for any scheme X we encounter, we write X instead of X (Fq ). PROPOSITION

6.1

We have |A G (s, r, k)| = |Gr(k, s)|

X

C(s, r, k, j)|A G (k, j, k)|.

j

Proof Write W = Fqs . For every map f ∈ W V , let h f i denote the span of the f (vi ). The map (Q, f ) 7→ h f i fibers the set A G (s, r, k) over Gr(k, s). The fiber over a subspace U ⊂ W is then the set A G (s, r, U ) of (Q, f ) ∈ A G (s, r, k) such that h f i = U . The transitivity of the GLs action on Gr(k, s) shows that the fibers all have the same number of points. Thus, for any given U , #A G (s, r, k) = #Gr(k, s) · #A G (s, r, U ).

(6.1)

Now let A G (s, r, U, j) be the set of (Q, f ) ∈ A G (s, r, U ) such that Q|U has rank j. This decomposes A G (s, r, U ) into disjoint subsets. Consider the map pU : A G (s, r, U, j) → A G (U, j, k)

MOTIVES AND A CONJECTURE OF KONTSEVICH

165

given by (Q, f ) 7→ (Q|U , f ). The fiber of pU above a given (Q, f ) ∈ A G (U, j, k) is C Q (s, r, k, j). Thus #A G (s, r, U, j) = C(s, r, k, j) · #A G (k, j, k).

(6.2)

Summing over all the j in equation (6.2) and substituting the result into equation (6.1), we obtain the desired result. PROPOSITION

6.2

We have |A G (s, r, k)| = |Gr(r, s)|

r X

q l(s−r ) |Homk−l (n − l, s − r )||A G (r, r, l)|.

(6.3)

l=0

Proof Write W = Fsp , and let 9 : A G (W, r, k) → Gr(s − r, W ) be the map associating to every (Q, f ) the kernel ker Q of Q. The fiber of 9 over a subspace U ⊂ W is the set A G (W, r, k)U consisting of all (Q, f ) ∈ A G (W, r, k) with Q|U = 0. The transitivity of the action of GL(W ) on Gr(s − r, W ) then shows that #A G (W, r, k) = #Gr(s − r, W ) · #A G (W, r, k)U .

(6.4)

Let T = W/U , and let π : W → T be the quotient map. Q reduces in an obvious way to a form Q on T . In fact, Q 7→ Q is a one-to-one correspondence between bilinear forms on W with kernel U and nondegenerate bilinear forms on T . Now stratify A G (W, r, k)U by the dimension d of hπ ◦ f i. The stratum corresponding to d = l maps to A G (T, r, l) by sending (Q, f ) to (Q, π ◦ f ). The fiber above a pair (Q, g) is identified with the set of maps f : V → W such that π ◦ f = g and h f i is of dimension k. It is now elementary linear algebra to verify that the number of such f ’s is given by q l(s−r ) Homk−l (n − l, s − r )(Fq ). (6.5) To see this, pick a splitting W = T ⊕ U , and visualize any f as an (n × s)-matrix whose upper (n × r )-rectangle agrees with g. This upper rectangle is then a matrix of rank l, and, without loss of generality, we can assume that the first l columns are linearly independent. It is then easy to check that f has rank k if and only if the bottom right ((n − l) × (s − r ))-rectangle has rank k − l. Thus this bottom right rectangle can be chosen to be an arbitrary matrix in Homk−l (n − l, s − r ). The bottom left (l × (s − r ))-rectangle can then be arbitrarily chosen among q l(s−r ) possible matrices. Now, putting (6.4) and (6.5) together, we obtain the desired result.

166

BELKALE and BROSNAN

It is worth recording an important special case of Proposition 6.2. 6.3 We have |A G (s, r, s)| = q r (s−r ) |Gr(r, s)||Homs−r (n − r, s − r )||A G (r, r, r )|. COROLLARY

Proof To get a nonzero contribution corresponding to l in Proposition 6.2, we need (1) r ≤ s, (2) l ≤ r, (3) l ≤ k, (4) l ≥ k + r − s. In the case of the corollary, s = k, so we get l ≤ r and l ≥ r . Hence l = r , and the formula reduces to exactly the above. We now give a reduction relating the incidence schemes of DG to those of G. PROPOSITION

6.4

We have |A DG (s, r, k)| = q k |A G (s, r, k)| + (q s − q k−1 )|A G (s, r, k − 1)|. Proof Let W = Fqs . Let f (V (DG)) → W with h f i a k-dimensional subspace. The span of f |V (G) is either a k-dimensional or a (k−1)-dimensional subspace. If {v} = V (DG)− V (G), counting the possibilities for f (v) proves the proposition. 7. The module of a graph For a simple graph G with n vertices, let M(G) be the R-submodule of CMot generated by the |A G (s, r, k)|. Let M(G)t be the submodule of M(G) generated by the |A G (s, r, k)| for s ≤ t. Proposition 6.1 shows that |A G (s, r, k)| ∈ M(G)k . Thus we have a finite filtration M(G) = M(G)n ⊃ M(G)n−1 ⊃ · · · ⊃ M(G)0 = R . The goals of this section are to compute the structure of M(G) and to show that, in fact, M(G) ⊂ CGraphs∗ . To do this, we introduce three special schemes: S K G (s) = A G (s, s, s), JG (s) = k A G (s, s, k), and HG (s) = A G (n, s, n). Note that JG (s) consists of the scheme of all pairs (Q, f ) ∈ A G (s) with Q ∈ Symss ; that is, there is no restriction on the rank of f . Note also that K G (n) = HG (n).

MOTIVES AND A CONJECTURE OF KONTSEVICH

167

7.1 |A G (s, r, k)| ∈ M(G)d for d = min(s, r, k). M(G)t is spanned as an R-module by the |K G (s)| for s ≤ t. M(G)t is spanned as an R-module by the |JG (s)| (resp., by HG (s)) for s ≤ t.

PROPOSITION

(a) (b) (c)

Proof For (a) and (b), apply Proposition 6.1 to obtain an expression for |A G (s, r, k)| as a Z[q]-linear combination of terms of the form |A G (k, j, k)| with j ≤ min(r, k) ≤ s. Then apply Corollary 6.3 to obtain an expression for each |A G (k, j, k)| as a Z[q]linear combination of terms of the form |K G ( j)|. For (c), to see that the |JG (s)| span, note that (a) implies that |JG (s)| ≡ |K G (s)| modulo M(G)s−1 . To see that the |HG (s)| span, use the fact that |HG (s)| = σ |K G (s)| for σ ∈ S, a consequence of Corollary 6.3. Our interest in the HG (s) is based on the following lemma, which allows us to compare the |HG (s)| to the |H DG (s)|. The lemma is essentially a translation of [23, Th. 5.1] into our language. As there are two graphs involved in the lemma, we write n G for the cardinality of V (G). 7.2 For r ≤ n G + 1, LEMMA

|H DG (r )| = aG (r )|HG (r )| + bG (r )|HG (r − 1)| + cG (r )|HG (r − 2)|

(7.1)

with aG (r ) = q n G +r (q n G +1 − 1), bG (r ) = q n G +r −1 (q n G +1 − 1)(q − 1), cG (r ) = q n G (q n G +1 − 1)(q n G +1 − q r −1 ), all polynomials in S. Proof By Proposition 6.4, |A DG (n G + 1, r, n G + 1)| = (q n G +1 − q n G )|A G (n G + 1, r, n G )|. Now applying Proposition 6.1 to |A G (n G + 1, r, n G )| and expanding out |Gr(n G , n G + 1)| in terms of q gives the result. The polynomials aG , bG , and cG in the lemma are clearly in S as long as they are nonzero. Inspection shows that this is the case under the assumption that r ≤ n G + 1.

168

BELKALE and BROSNAN

There is a simpler identity relating JG (s) to J DG (s). 7.3 We have |J DG (s)| = q s |JG (s)|. PROPOSITION

Proof The obvious map J DG (s)(Fq ) → JG (s)(Fq ) restricting f from V (DG) to V (G) has fiber Fqs . A direct consequence of Propositions 7.3 and 7.1(c) is the following. PROPOSITION 7.4 We have M(DG) = M(G).

We are now ready to prove the main theorem of this section. 7.5 M(G) is equal to the R-module spanned by the functions |Z D k G | for k ≥ 0. In particular, M(G) ⊂ CGraphs∗ . THEOREM

Proof For the proof, let N (G) be the R-module spanned by the functions |Z D k G | for k ≥ 0. Since |K G (n G )| = |Z G ||GLn G |, |Z G | ∈ M(G). Thus it follows from Proposition 7.4 that N (G) ⊂ M(G). To prove that M(G) ⊂ N (G), we use Lemma 7.2 and an inductive argument. By Proposition 7.1(c), it is enough to show that |HG (s)| ∈ N (G) for all s. Since |HG (n G )| = |K G (n G )|, this is obvious for s = n G . Now, by Lemma 7.2, |H DG (n G + 1)| = bG |HG (n G )| + cG |HG (n G − 1)|

(7.2)

with bG , cG ∈ S. (The first term on the right-hand side of (7.1) vanishes because HG (n G + 1) is empty.) We know that |H DG (n G + 1)| and |HG (n G )| are in N (G). Thus |HG (n G − 1)| ∈ S. We then assume inductively that |HG (n G − i)| ∈ N (G) for all i ≤ a and for all graphs G. Another application of Lemma 7.2 shows us that

H DG n G − (a − 1) = aG HG n G − (a − 1) + bG |HG (n G − a)|
+ cG HG n G − (a + 1) . (7.3) By induction, the left-hand side and the two first terms on the right-hand side are in N (G). Thus, as cG ∈ S, |HG (n G − (a + 1))| ∈ N (G) as well.

MOTIVES AND A CONJECTURE OF KONTSEVICH

169

8. Matroid theory A matroid M consists of a finite set E called the edges of the matroid and a rank function ρ : 2 E → N satisfying the following axioms: (1) for X ⊂ E, ρ(X ) ≤ #X ; (2) for X ⊂ Y ⊂ E, ρ(X ) ≤ ρ(Y ); (3) for any X , Y ⊂ E, ρ(X ∪ Y ) + ρ(X ∩ Y ) ≤ ρ(X ) + ρ(Y ).

(8.1)

The integer ρ(E) is said to be the rank of the matroid. Matroids were introduced by H. Whitney [29] as a simultaneous generalization of matrices and graphs. An excellent modern reference for matroid theory is [21]. 8.0.1. Representability A matroid M of rank r is said to be representable over a field K if there is a function f : E → K r such that the dimension of the span of the set f (X ) is equal to ρ(X ) for all X ⊂ E. 8.0.2. Matroids from matrices For every finite subset E ⊂ K s there is naturally a matroid M representable over K given by setting ρ(X ) = dimhX i for every X ⊂ E. π Let K n+1 − {0} 7→ Pn (K ) be the natural map taking a nonzero vector in v ∈ K n+1 to the line K v. Suppose that E ⊂ Pn (K ). Then any set-theoretic splitting σ : Pn (K ) 7→ K n+1 gives a subset σ (E) of K n+1 and thus defines a matroid. It is easy to see that this matroid is independent of the splitting σ . Thus, since such splittings always exist, E defines a matroid. 8.0.3. Representation schemes For any matroid M and a locally free sheaf W over a base S, let X (M, W ) be the subscheme of Fun(E, W ) consisting of all f whose restrictions to Fun(T, W ) lie in Funρ(X ) (T, W ) for all T ⊂ E. This is the scheme of representations of M in W . When S = Spec Z and W = O Ss , we write X (M, s) for X (M, W ) as in the introduction. For a field K , X (M, s)(K ) is the set of all maps f : E → K s such that dimh f (T )i = ρ(T ) for all T ⊂ E. That is, X (M, s)(K ) is the set of all representations of M in K s . When r is the rank of M, we write X (M) for X (M, r ). X (M)(K ) is nonempty if and only if M is representable over K . Clearly, X (M, s)(K ) is nonempty if and only if s ≥ r and X (M)(K ) is nonempty. Definition 8.1 Let CMatroids be the R-module generated by all functions of the form |X (M)|.

170

BELKALE and BROSNAN

Remark 8.2 When M is a rank r matroid, it is easy to see that |X (M, s)| = |Gr(r, s)||X (M)|. Thus CMatroids is the same as the R-module generated by all functions of the form |X (M, s)|. In the next section, we show that CMatroids ⊂ CGraphs∗ . 9. A counterexample to Kontsevich’s conjecture Let G be a graph, let V be the set of its vertices, and let U ⊂ 2V . A function π : U 7→ N is called a partially defined rank function for V . Notice that the data of a partially defined rank function π determines U = dom(π). Associated to every such function we have a scheme defined as follows. Definition 9.1 JG (s, π ) is the scheme of all (Q, f ) ∈ JG (s) such that f |H has rank ρ(H ) for all H ∈ dom π . THEOREM 9.2 For every G and every partially defined rank function π for V (G), |J (s, π )| ∈ CGraphs∗ .

Proof The proof is by induction on the cardinality of dom(π ). If dom(π ) is empty, JG (s, π) = JG (s). Thus the result follows from Theorem 7.5. Now assume that the result holds for all graphs G and all π such that #dom π ≤ a. Let W ⊂ 2V be a set of subsets with a+1 elements, let H ∈ W , and let U = W −{H }. Let π : U → N be a partially defined rank function, and let πi : W → N be the extension of π to W such that πi (H ) = s − i. Clearly, any partially defined rank function with domain W is of the form πi for some π : U → N and some i ∈ [0, s]. Now for each t ∈ N we define a graph G t as follows. G t is the graph obtained from G by adjoining t disjoint vertices y1 , . . . , yt and connecting each of the yi by edges only to the vertices in H . Thus V (G t ) = V (G) ∪ Y , where Y = {y1 , . . . , yt }, and E(G t ) = E(G) ∪ {ehy }h∈H . y∈Y

Since V (G) ⊂ V (G t ), U ⊂ 2V (G t ) , we can consider π as a partially defined rank function for V (G t ).

MOTIVES AND A CONJECTURE OF KONTSEVICH

171

The result follows from the equation |JG t (s, π)| =

s X

q ti |JG (s, πi )|.

(9.1)

i=0

To see that equation (9.1) holds, note that we can stratify the Fq -points of JG t (s, π) according to the dimension of the span of f (H ). Let JG t (s, π)i be the stratum where this dimension is s − i. This stratum maps to JG (s, πi ) by restricting f from V (G t ) to V (G). The fiber of the map above any point (Q, f ) is an affine space Ati . This is because the only condition on the f (yi ) is that they be orthogonal to the span of f (H ). Thus, as the bilinear form Q is always nondegenerate, they must lie in a linear subspace of dimension i. To complete the proof, note that by varying the t from zero to s we obtain a system of equations for the |JG (s, πi )| in terms of the |JG t (s, π)|. Solving this system for the JG (s, πi ) using Cramer’s rule, we have to invert a Vandermonde determinant that lies in S. Thus, as we assumed by induction that |JG t (s, π)| lies in CGraphs∗ , it follows that each |JG (s, πi )| lies in CGraphs∗ as well. This leads to the following theorem. THEOREM 9.3 We have CMatroids ⊂ CGraphs∗ .

Proof Let G be a discrete graph (i.e., E(G) is empty). In this case, if π is a partially defined rank function, then |J (s, π)| = |Symss ||L(s, π)|, (9.2) s where L(s, π) is the scheme consisting of all f ∈ Fun(V, OSpec Z ) such that f res stricts to Funπ(H ) (H, OSpec Z ) for all H ∈ dom(π). To see this, note that the definition of J (s, π ) makes it clear that Q does not enter in the definition of the J ’s for the discrete graph. And only the vertex set V is needed for the definition of the L’s since G is discrete. As Symss ∈ S, it follows that the L’s are all in CGraphs∗ . Now note that, if M is a matroid, with rank function ρ : 2 E → N, then X (M, s) = L(s, ρ).

It is now possible to see directly that Conjecture 0.4 and thus Conjecture 0.1 are false. Let M be the Fano matroid. This is a rank 3 matroid whose edge set E is the set P2 (F2 ). It is representable over a field Fq if and only if 2|q (see [28, Chap. 9]). Thus the function |X (M)| is supported on the set of q such that 2|q. It follows

172

BELKALE and BROSNAN

that |X (M)| cannot be a rational function. And this contradicts Conjecture 0.4 by Theorem 9.3. Remark 9.4 By unravelling the induction used in the proof Theorem 9.3, we can be a bit more specific about the nature of the counterexamples arising from the Fano matroid. The first step is to use the Fano to find a graph G such that JG (3) is not polynomially countable. Let V = {1, . . . , 7}, and view V as the vertex set of a disjoint graph. Let F be the set of functions from 2V to {0, 1, 2, 3}. For each such function φ ∈ F , we construct a bipartite graph G φ as follows. For every H ⊂ V , we add φ(H ) new vertices to V , and we connect each of these new vertices by an edge to the P vertices of H . In the end, we have a bipartite graph with H ⊂V φ(H )+7 vertices and P H ⊂V φ(H )|H | edges. An inspection of the induction from the proof of the theorem shows that, if we range over all φ ∈ F , we are guaranteed to produce a graph G φ such |V | that JG φ (3) is not polynomially countable. Unfortunately, |F | = 42 = 2256 ; thus we are far from having an explicit graph. Also, note that the set of graphs produced from a given matroid depends only on the order of the matroid. Thus we would obtain the same set of graphs from any matroid of order 7. To produce an explicit counterexample to Kontsevich’s conjecture, we must work even harder. Once we find a G such that JG (3) is not polynomially countable, we know that Z D k G is not polynomially countable for some k. The induction used to prove Theorem 7.5 allows k to range from zero to the number of vertices of G. Thus, if we use a G φ as above, we could have as many as 3 · 27 + 7 graphs to search through. However, once we do have a graph G with Z G not polynomially countable, G o is a counterexample to Conjecture 0.4. Thus we learn that there is a bipartite graph whose complement is a counterexample to Conjecture 0.4. Set C = (G o )∗ . C is then a counterexample to Stanley’s version of Kontsevich’s conjecture, Conjecture 0.3. To find a counterexample to Kontsevich’s original conjecture, we have one more step that, following Stanley’s proof of the equivalence of the two conjectures, involves searching through all graphs of the form (C/S) − T with S ⊂ E(C) and T ⊂ E(C/S). Clearly, this is also a very large number of graphs. Thus the interesting problem of finding an explicit counterexample to Kontsevich’s conjecture is totally open. 10. Mn¨ev-Sturmfels universality Our objective in this section is to show that Matroids = Mot and thus that CGraphs∗ = CMot , completing the proof of Theorem 0.5. This follows from the known results on the matroid representation problem. We saw in Section 9 that |L(k, π )| were in CGraphs∗ even if π is only partially

MOTIVES AND A CONJECTURE OF KONTSEVICH

173

defined. It suffices therefore to show that the R-module generated by all functions of the form |L(k, π)| is all of CMot . This was in essence proved by N. Mn¨ev [19], [20] as the unoriented matroid component of a more difficult theorem concerning the representation spaces of oriented matroids∗ (see also [11], [22]). It was independently proved by J. Bokowski and B. Sturmfels [3], [26]. Moreover, the idea of the proof using K. von Staudt’s “algebra of throws” goes back at least to [16] (see [15] for an enlightening explication). However, as we have been unable to extract a proof of the exact statement that we need from the literature, we give a sketch of the proof in our context. 10.1 (Mn¨ev, Sturmfels) If X is a quasi-projective scheme of finite type over Z, then there is a set V , a set of subsets W of V , a function π : W → Z, and an element σ ∈ S so that THEOREM

σ |X | = |L(3, π)|. Remark 10.2 (1) The theorems in matroid theory are not in such a direct form because, in matroid theory, we are committed to declaring the rank of all the subsets of V . Our partially defined π does not have this problem. By inclusion-exclusion principles, the R-module generated by all functions of the form |L(k, π)|, where π may be only partially defined, is the same as the R-module generated by all functions of the form |L(k, π )|, where π is defined on all subsets of V . (2) Note that any scheme of finite type/Z is a finite disjoint union of quasiprojective schemes/Z. Proof The proof follows essentially from the following observations. (1) Four elements in P 2 such that any three are linearly independent can, by a unique automorphism of P 2 in PGL(2), be assumed to be (1, 0, 0), (0, 1, 0), (0, 0, 1), and (1, 1, 1). (2) If given two points (x, 0, 1) and (x 0 , 0, 1) on the X -axis, then, by just drawing lines through the four points above and these two points, we can locate (x + x 0 , 0, 1), (x x 0 , 0, 1), (−x, 0, 1). The intersection of two lines is a point that lies on both lines. We can code this using matroids by introducing a new point and adding linear dependence conditions on this point and points on the two lines. These constructions can be found, for example, in the proof of [3, Th. 2.2]. ∗ Mn¨ ev showed that any variety over a prime field k = Q or F p has a nonempty open subvariety that is isomorphic to the variety of projective representations of a rank 3 matroid with four fixed edges in the matroid mapping to four fixed points in P2 (k). Using this statement, Theorem 10.1 can be proved by Noetherian induction.

174

BELKALE and BROSNAN

(3) Iterating these constructions, given (x1 , x2 , . . . , xn ) we can determine the points ( f (x1 , x2 , . . . , xn ), 0, 1), where f is a polynomial with integer coefficients, by just drawing lines starting from the configuration of the four given points and the points (xi , 0, 1). Setting f (x1 , . . . , xn ) either equal to zero or not equal to zero is just another spanning condition, a condition on whether
f (x1 , x2 , . . . , xn ), 0, 1 , (0, 0, 1) is linearly independent or not. (4) The cone over any quasi-projective scheme/Z can be written as a set of equalities and a set of nonequalities in a finite set of variables (x1 , . . . , xn ). Note that we can also have conditions of the form n = 0 in the list. (5) From the previous considerations we obtain a set S = {P1 , P2 , P3 , P4 , Q 1 , . . . , Q n , T1 , . . . , Tl } of cardinality (say) c, where the Pi correspond to (1, 0, 0), (0, 1, 0), (0, 0, 1), and (1, 1, 1). The Q j correspond to (x j , 0, 1), and the T ’s to intermediate points in the constructions. The preceding discussion also gives a partially defined rank function π on this set (this function and the set S are determined by the equations and nonequalities defining the scheme X ) and a map p : L(k, π ) 7→ (PZ2 )4 by the image of the Pi , so that the fiber over ((1, 0, 0), (0, 1, 0), (0, 0, 1), (1, 1, 1)) is equal to Gcm ×Z X as a scheme over Z. We can identify the image of p with PGL(3). Moreover, over this image, p is a Zariski locally trivial fibration. Thus we have [L(k, π )] = [PGL(3)][Gcm ][X ].

(10.1)

11. Forests In this section, we prove that |JG (s)| ∈ Z[q] whenever G is a forest. It follows that M(G) = Z[q] for such graphs. To do so, we need to introduce two operations on graphs. Let v ∈ V (G). We obtain a graph Iv (G) by adding one edge e connected to v and one new vertex w connected to e. That is, we insert an edge at v. Clearly, a graph is a tree if and only if it can be obtained from the graph with one vertex by successive applications of Iv for various v. A graph is a forest if and only if it can be obtained from the empty graph ∅ by successive applications of Iv and the operation D. We write Dn for the graph D n ∅. We define Rv to be the graph obtained from G by deleting v and all edges meeting it. Note that if G is a forest and if v is any vertex in G, then Rv G is also a forest. 11.1 Let G be a graph with v ∈ V (G). We have the following: (a) |J Dn (s)| = q ns |Symss |, THEOREM

MOTIVES AND A CONJECTURE OF KONTSEVICH

(b) (c)

175

|J DG (s)| = q s JG (s), |J Iv (s)| = q s−1 (JG (s) + (q − 1)J Rv G (s)).

Proof Part (b) is a restatement of Proposition 7.3. For (a), assume first that n = 0. Then, tracing through the definitions, one sees that J∅ (s) = Symss . The rest of (a) follows by induction from (b). For (c) we work over Fq and consider the map π : J Iv (G) (s) → JG (s) given by (Q, f ) 7→ (Q, f |V (G) ). The fiber of π above a point (Q, g) ∈ JG (s) depends on whether g(v) is zero or not. Let JG0 (s) (resp., JG× (s)) be the set where g(v) = 0 (resp., g(v) 6= 0). Above a point (Q, g) ∈ JG0 (s), the fiber of π has q s points. Above a point (Q, g) ∈ JG× (s), the fiber has q s−1 points since Q is nondegenerate. Thus |J Iv (G) (s)| = q s−1 |JG× (s)| + q s |JG0 (s)|.

(11.1)

The result now follows from the observation that |JG0 (s)| = |J Rv G (s))|. 11.2 For F a forest, |Z F | ∈ Z[q]. COROLLARY

Proof An easy induction using Theorem 11.1 shows that |J F (s)| ∈ Z[q] for any s. Thus M(F) = R . It follows from Theorem 7.5 that |Z F | ∈ R . But this implies that |Z F | ∈ Z[q] by Proposition 1.3. The next corollary follows from Theorem 11.1 and the results in Section 3.1. COROLLARY 11.3 Let F be a forest with r vertices contained in a complete graph K s . Let G = K s − F. o | ∈ Z[q]. (1) We have |Z G (2) If s > r , then |X G | ∈ Z[q].

12. Geometric motives The purpose of this section is to develop the necessary machinery for showing that the results stated thus far in the context of combinatorial motives continue to hold in the geometric context. The main tool we need is an efficient apparatus for converting fibrations of the type used to prove the results of Section 6 into formulas in GeoMot . The fibrations that automatically yield such formulas are called piecewise Zariski fibrations in the motivic integration literature (see [7]). If F →X →Y

176

BELKALE and BROSNAN

is a piecewise Zariski fibration, then one obtains a formula [X ] = [F][Y ]. To write such a formula, one must know F as a scheme. It is useful for us to have a slightly more general notion that we call motivic fibrations, which allow us to write formulas where the fiber F is known only as a motive. To say precisely what we mean by a motivic fibration, we must first develop the notion of motives over a general Noetherian base. This is not much more difficult than the notion of a motive over Z, and we see that it allows more flexibility. We also see that there are tools for reducing questions about motives over a base to questions about motives over fields. Let S be a Noetherian scheme. (We make the standing assumption that all base schemes are Noetherian.) We write GeoMot+ motives over S. As an S for the ring of f + abelian group, GeoMot S is generated by the symbols [X →S] for X a scheme of finite type over S under the following relations: f g (a) [X → S] = [Y → S] if X ∼ = Y as S-schemes; (b)

if V is closed in X and U = X − V , then f |U

f

f |V

[X → S] = [U → S] + [V → S]. f

When the map f is clear, we write [X ] S or simply [X ] for [X → S]. If S = Spec R, we also write [X ] R . It follows from (a) and (b) that [X ] S = 0 when X is the empty scheme. A structure of a commutative ring is induced on GeoMot+ S by setting [X ][Y ] = [X × S Y ]. It is easy to check that this operation is well defined with respect to the relations (a) and (b). The unit in this ring is [S] S . Note that [X ] S = [X red ] S . This is because X red is a closed subscheme of X whose complement is empty. Using this fact and Noetherian induction, we obtain a well-defined class [C] S for any constructible subset C ⊂ X of an S-scheme. 12.1. Base change Suppose that u : S → T is a morphism. Base change then provides a ring homomor+ phism u ∗ : GeoMot+ T → GeoMot S explicitly given by [X ]T 7→ [X ×T S] S .

(12.1)

Suppose that S is of finite type over T . Then there is a group homomorphism + u ∗ : GeoMot+ S → GeoMotT given by f

u◦f

[X → S] 7→ [X → T ].

(12.2)

The ring of motives over S is “topological,” that is, depends only on the reduced scheme structure of S.

MOTIVES AND A CONJECTURE OF KONTSEVICH

177

LEMMA 12.1 + The map i : Sred → S induces an isomorphism i ∗ : GeoMot+ S → GeoMot Sred with + + inverse given by i ∗ : GeoMot Sred → GeoMot S .

Proof The lemma is a consequence of the fact that (X × S Sred )red = X red . 12.2. Motives over finite fields Suppose that k is a finite field. Then there is a map GeoMot+ k → N given by [X ] 7→ #X (k). If S = Spec Z, there is an evaluation map from GeoMot+ S to the set of functions from Q to Z. Example 12.2 Let X and S be two copies of the multiplicative group scheme Gm , and let f : X → S be the squaring map. Then [X ]Spec Z = [S]Spec Z but [X ] S 6= [S] S . To see this, let k be any finite field, and let η : Spec k → Gm be the map corresponding to the point 1 ∈ Gm (k). Then η∗ [X ] S has two points while η∗ [S] S has only one. 12.3. Zariski fibrations f

A map X → Y of schemes over a base S is said to be a Zariski fibration with fiber F if there is a covering of Y by open sets Yi such that X ×Y Yi ∼ = F × S Yi .

(12.3)

We remark that the pullback of a Zariski fibration with fiber F is also a Zariski fif

g

bration with fiber F. Also, if X → Y and Y → Z are Zariski fibrations over S with fibers A and B, respectively, then g ◦ f is a Zariski fibration with fiber A × S B. 12.3 f Let u : Y → S be a structure map, and suppose that X → Y is a Zariski fibration over S with fiber F. Then [X ]Y = u ∗ [F] S . (12.4) PROPOSITION

Proof Let Yi be any open cover of Y . Set X i = X ×Y Yi . As X i is an open cover of X , P [X ]Y = [X i ]Y . P Now pick an open cover {Yi } so that X i ∼ = F × S Yi . Then [X ]Y = [F × S Yi ]Y . P On the other hand, [F × Y ]Y = [F × S Yi ]Y . Thus [X ]Y = u ∗ [F] S .

178

BELKALE and BROSNAN

12.4. Motivic fibrations f

u

For a sequence of morphisms X → Y → S, we say that f is a motivic fibration over f

S with fiber A ∈ GeoMot S if [X → Y ]Y = u ∗ A for A ∈ GeoMot S . f

A Zariski fibration is a motivic fibration. If X → Y is a morphism and if V is a f |V

f |U

closed subset of X with complement U such that V → Y and U → Y are motivic f

fibrations with fibers A and B, respectively, then X → Y is a motivic fibration with fiber [A] + [B]. The property of being a motivic fibration is invariant under base change. That is, if X → Y is a motivic fibration over S with fiber A and if Y 0 → Y is any map, then X ×Y Y 0 → Y 0 is a motivic fibration with fiber A. 12.4 u If X → Y → Z → S is a sequence of morphisms and if both f and g are motivic fibrations over S with fibers A and B, respectively, then g ◦ f is a motivic fibration with fiber A × B. PROPOSITION f g

Proof P P There are S-schemes Ai and B j such that αi [Ai ] = A and β j [B j ] = B with αi , β j integers. By assumption, we have [X ]Y = (u ◦ g)∗ A and [Y ] Z = u ∗ [B]. P P Thus [X ]Y = [Ai × S Y ]Y . It follows that [X ] Z = αi [Ai × S Y ] Z , and thus P [X ] Z = i j αi β j [Ai × S B j × S Z ]. 12.5 Let M be a motive over a reduced and irreducible base S. If M K (S) = 0, then there is a nonempty open set U ⊂ S such that MU = 0 (where K (S) is the field of fractions of the coordinate ring of S, also called the function field of S, and M K (S) = MSpec K (S) ). LEMMA

Proof Set K = K (S). The lemma follows from two considerations. (1) If f : X → Y is a map of schemes of finite type over S such that f K is an isomorphism, then there is a nonempty open set U ⊂ S such that fU is an isomorphism. (2) If X K = A ∪ B with A a closed subscheme and B its complement, then there is a nonempty open subset U of S and two disjoint schemes A0 and B 0 with A0 closed and B 0 its complement such that A = A0K and B = B K0 . 12.6 Let f : X → Y be a map of two schemes of finite type over a base S. Suppose that there exists a motive M over S such that, for every every field-valued point η ∈ Y (K ), PROPOSITION

MOTIVES AND A CONJECTURE OF KONTSEVICH

179

f K : X η → Spec K is a motivic fibration with fiber M K . Then f : X → Y is a motivic fibration with fiber M. Proof We can assume, without loss of generality, that Y is reduced and irreducible. Let K be the function field of Y . Then the hypothesis of the proposition implies that [X K ] K − M K = 0. Thus there exists a nonempty open set U ⊂ Y such that [X U ]U − MU = 0. Let V = Y −U . Then f V : X V → V again satisfies the hypothesis of the proposition; hence the proposition follows by Noetherian induction. 13. Vector bundles and extensions The purpose of this section is to show that the formulas of Section 5 involving extensions of quadratic forms continue to hold in a motivic setting with q replaced with L. Once this is done, we obtain a motivic version of MacWilliams’s formula for Symrn . We also build the groundwork for motivic versions of the formulas in Section 6. A vector bundle M of rank m over a base S is a locally free coherent sheaf of rank m on S. We consider the problem of lifting symmetric bilinear forms on a subbundle V ⊂ W to W . Since this has to be done by induction, we have to keep geometric control in such arguments. A subsheaf N ⊂ M is a subbundle if the quotient M/N is locally free. 13.1 Let 0 $ N $ M be a subbundle of a bundle M, over a scheme S. For a vector bundle U of rank u on S and for integers r ≤ q, let X be the scheme over S representing the functor  T φ ∈ Hom(MT , UT ) rk(φ|N T ) = r, rk(φ|MT ) = q . (13.1) PROPOSITION

Then the natural map X → Homr (N , U )

(13.2)

is a motivic fibration over S with fiber Lr (m−n) Homq−r (m − n, u − r ),

(13.3)

where m = dim(M) and n = dim(N ). Proof Let k be a field with a map η : Spec k → Homr (N , U ). The map η corresponds uniquely to a morphism φ : Nk → Uk of rank r . The fiber X = Homρ (F, U )η is then the scheme of all extensions ψ : Mk → Uk of rank q such that ψ|Nk = φ. Pick a basis e1 , . . . , en of Nk , and extend it to a basis e1 , . . . , em of Mk . Let C be the k

180

BELKALE and BROSNAN

vector space spanned by the en+1 , . . . , em . Let f 1 , . . . , fr be a basis for φ(Nk ), and extend it to a basis f 1 , . . . , f p of Uk . Write U1 for φ(Nk ), U2 for the k vector space spanned by the f i for i > r , and pri : U → Ui for respective projections. An extension ψ of φ has rank q if and only if the restriction of pr2 ◦ ψ to C has rank q − r . It follows that there is an isomorphism X → Hom(C, U1 ) × Homq−r (C, U2 )

(13.4)

ψ 7→ (pr1 ◦ ψ|C, pr2 ◦ ψ|C).

(13.5)

given by

We now consider the “dual” version of Proposition 13.1. PROPOSITION 13.2 Let 0 $ N $ M be a subbundle of a bundle M, over a scheme S. For a vector bundle U of rank u on S and for integers r ≤ q, let X be the scheme over S representing the functor  T φ ∈ Hom(UT , MT ) rk(φ : UT → MT ) = q, rk(φ : UT → MT /N T ) = r . (13.6) Then the natural map

X → Homr (U, M/N )

(13.7)

is a motivic fibration over S with fiber Lr n Homq−r (n, u − r ),

(13.8)

where m = dim(M) and n = dim(N ). Proof The proposition is proved by taking duals in the proof of Proposition 13.1. 13.1. Extensions of bilinear forms For any three integers d, r1 , and r2 , set   Lr1 , r1 = r2 ,    Lr2 − Lr1 , r2 = r1 + 1, γ (d, r2 , r1 ) = d+1 r +1 1  L −L , r2 = r1 + 2,     0 otherwise.

(13.9)

MOTIVES AND A CONJECTURE OF KONTSEVICH

181

LEMMA 13.3 Let 0 $ N $ M be a subbundle of rank n of a bundle M of rank n + 1, over a scheme S. For integers r ≤ q, let Z denote the representable functor  (13.10) T Q ∈ Sym(M) rk(Q M ) = q, rk(Q N ) = r .

Then the natural map Z → Symr N

(13.11)

is a motivic fibration over S with fiber γ (n, q, r ). Proof Suppose that η : Spec k → Symr (N ) is a map from the spectrum of a field to Symr N corresponding to a symmetric bilinear form Q of rank r on Nk . Set X = Z ×Symr N Spec k. (13.12) Then X is the scheme of all bilinear forms R on Mk of rank q such that R|Nk = Q. Let Y be the scheme of all bilinear forms R on Mk such that R|Nk = Q, and, for each i, let Yi be the closed subscheme of Y consisting of forms of rank less than or equal to i. Then X = Yq − Yq−1 . The lemma follows from the following set of isomorphisms, which can each be proved using elementary linear algebra: (a) Y = Yr +2 ∼ = An+1 , (b) Yr +1 ∼ = Ar +1 , ∼ (c) Yr = Ar . Now we define geometric analogues 0(d2 , r2 , d1 , r1 ) of the C(d2 , r2 , d1 , r1 ) by requiring that (a) 0(d1 + 1, r2 , d1 , r1 ) = γ (d1 , r2 , r1 ), (b) the following recursion is satisfied: 0(d2 , r2 , d1 , r1 ) =

2 X

0(d2 , r2 , d1 + 1, r1 + j)0(d1 + 1, r1 + j, d1 , r1 ).

j=0

(13.13) Clearly, 0(d1 + 1, r2 , d1 , r1 ) is a polynomial in L and
ev 0(d1 + 1, r2 , d1 , r1 ) = C(d1 + 1, r2 , d1 , r1 ).

(13.14)

PROPOSITION 13.4 Let 0 $ N $ M be a subbundle of dimension d1 of a bundle M of dimension d2 , over

182

BELKALE and BROSNAN

a scheme S. For integers r1 ≤ r2 , let Z denote the representable functor  T Q ∈ Sym(M) rk(Q M ) = r2 , rk(Q N ) = r1 .

(13.15)

Then the natural map Z → Symr1 N

(13.16)

is a motivic fibration with fiber 0(d2 , r2 , d1 , r1 ). Proof Let K be a field, and let η ∈ Symr N (K ) be a point corresponding to a bilinear form Q ∈ Symr1 N K . Let X be the variety of extensions of Q to a symmetric bilinear form on M K of rank r2 . By Proposition 12.6, it suffices to show that [X ] K = 0(d2 , r2 , d1 , r1 ). We prove the proposition by induction on d2 − d1 . If d2 ≤ d1 + 1, the result follows from Lemma 13.3. Otherwise, let P be a subspace of dimension d2 − 1 such that M K ⊃ P ⊃ N K . For a given q, let Yq be the scheme of extensions of Q to a symmetric bilinear form on P of rank q. Let X q be the locally closed subscheme of extensions Q ∈ X such that Q|P ∈ Yq . The X q stratify X into a disjoint union of locally closed subschemes. Let Z P be the scheme of all rank d2 forms on M K which restrict to rank q forms on P. By the inductive hypothesis, [Yq ] K = 0(d2 − 1, q, d1 , r1 ). But then note that the diagram / Yq Xq (13.17)  ZP

 / Symq P

is a pullback. Thus X q → Yq is a motivic fibration with fiber 0(d2 , r2 , d2 − 1, q). The proposition now follows from Proposition 12.4. We now obtain a motivic version of MacWilliams’s result counting the number of symmetric bilinear forms of a given rank and dimension over Fq . PROPOSITION 13.5 Equations (2.10) hold with L replacing q. That is, (Qs Q2s−1 n−i L2i − 1), 0 ≤ r = 2s ≤ n, i=1 L2i −1 · i=0 (L n [Symr ] = Qs Q2s L2i n−i − 1), 0 ≤ r = 2s + 1 ≤ n. i=1 L2i −1 · i=0 (L

(13.18)

MOTIVES AND A CONJECTURE OF KONTSEVICH

183

Proof MacWilliams’s proof (see [17, pp. 154 – 156]) relies only on the recursion |Symrn+1 | = q r |Symrn | + (q − 1)q r −1 |Symrn−1 | + (q n+1 − q r −1 )|Symrn−2 |. (13.19) Thus Proposition 13.5 follows from MacWilliams’s proof once this recursion is known to hold in GeoMot+ with L replacing q. This follows from Proposition 13.4. 14. Reduction formulas for GeoMot We begin with an inventory of what remains to be re-proved in the geometric context. Equations (2.7) – (2.9) were stated without proof in the combinatorial context. As they are easy, we leave them without proof in the geometric context. The three formulas in Section 6 are more difficult. We prove the geometric versions of Propositions 6.1 and 6.2. We leave to the reader the geometric version of Proposition 6.4, whose proof uses Proposition 13.1 and is similar to the proof of Proposition 6.2. The remaining formulas that need to be verified in GeoMot are Proposition 7.3 and equation (9.1). These we also leave to the reader. Once these formulas are checked to hold in GeoMot , Theorem 0.6(a) follows by pure algebra. To obtain Theorem 0.6(b), it must be checked that Theorem 11.1(c) holds in GeoMot with L replacing q. Again, we leave this to the reader. We need some notation concerning Grassmanians. If W is a vector bundle of over S, let π : Gr(r, W ) → S be the structure map from the Grassmanian of rank r subbundles of W to S. We write Sub for the universal subbundle and Quot for the universal quotient. Thus we have an exact sequence Sub  π ∗ W  Quot .

(14.1)

Suppose that W = O Ss . Then A G (W, r, k) = A G (s, r, k) × S. This can be easily checked by working through the functorial definition of A G (W, r, k). It follows that, if W now is any locally free sheaf of rank s, the map A G (W, r, k) → S is a motivic fibration with fiber A G (s, r, k) for any locally free sheaf W of rank s. 14.1 For any locally free sheaf W of rank s over a scheme S, X [A G (W, r, k)] S = [Gr(k, W )] S 0(s, r, k, j)[A G (k, j, k)]. PROPOSITION

(14.2)

j

Proof For every j, let X j be the subscheme of A G (W, r, k) representing the functor  T (Q, f ) ∈ A G (W, r, k)(T ) rk(Q|h f i) = j . (14.3)

184

BELKALE and BROSNAN

The X j stratify A G (W, r, k) into a disjoint union of locally closed subschemes. Let π : Gr(k, W ) → S be the structure map, and let Sub be the universal subbundle. There is then a map p j : X j → A G (Sub, r, k) (14.4) given functorially by sending a pair (Q, f ) ∈ X j (T ) to the pair (Q|h f i, f ). Let Z denote the scheme over Gr(k, W ) representing the functor  T Q ∈ Sym(WT ) rk(Q W ) = r, rk(Q Sub ) = j .

(14.5)

It is an exercise in chasing the functorial definitions of the various schemes involved to check that there is a pullback diagram as follows: Xj

/ A G (Sub, j, k)

 Z

 / Sym j (Sub)

(14.6)

From this it follows that [AG (W, r, k)] =

k X

0(s, r, k, j)[A G (Sub, r, k)].

(14.7)

j=0

The proposition follows from the fact that A G (Sub, r, k) fibers motivically over its base, Gr(k, W ) with fiber A G (k, j, k). 14.2 In GeoMotZ , COROLLARY

[A G (s, r, k)] = [Gr(k, s)]

k X

[0(s, r, k, j)][A G (k, j, k)].

(14.8)

j=0

We now give a version of Proposition 6.2 in GeoMot . PROPOSITION 14.3 Let W be a locally free sheaf of rank s on S. Let π : Gr(r, W ) → S be the canonical map, and let Quot be the canonical quotient of π ∗ W . We have

[A G (W, r, k)] S =

r X l=0

Ll(s−r ) [Homk−l (n − l, s − r )][A G (Quot , r, l)] S .

(14.9)

MOTIVES AND A CONJECTURE OF KONTSEVICH

185

Proof Let p : W → W/(ker Q) be the quotient map, and let Yl be the subscheme of A G (W, r, k) representing the functor  T (Q, f ) ∈ A G (W, r, k) rk( p ◦ f ) = l . (14.10) Let Z denote the scheme over Gr(k, W ) representing the functor (where V is the set of vertices of G)  T φ ∈ Hom(OTV , WT ) rk(φ : OTV → QuotT ) = l, rk(φ : OTV → WT ) = r . (14.11) It can then be checked that there is a pullback diagram Yl

/ A G (Quot , r, l)

 Z

 / Homl (O V , Quot) S

(14.12)

The proposition then follows from Proposition 13.2. 15. Periods We want to discuss briefly the implications on periods (integrals) which formed the basis for the original conjecture. The main difficulty is that we are working in the Grothendieck group of motives, and periods are not defined for these objects. We first show that there is a graph G such that the scheme YG has a cohomology group that is not mixed Tate. Then the periods of that cohomology group cannot be in the ring of multiple zeta values if we assume the following conjecture: if periods of a cohomology group (of an algebraic variety over Q) are multiple zeta values, then the cohomology group is mixed Tate. In this discussion, by cohomology we mean the pair of Betti cohomology with the mixed Hodge structure and algebraic de Rham cohomology with the comparison isomorphism. Note that the de Rham cohomology is defined over Q for varieties defined over Q. Consider the E-polynomials of Craw [6]. For X a projective scheme over Z, X X
E(X ) = (−1)k h p,q Hck (X C , C) u p v q , 1≤ p,q≤dim(X ) k≥0

where h p,q are the Hodge-Deligne numbers. A mixed Tate cohomology group has h p,q = 0 unless p = q. It is shown by Craw [6] that this function factors through GeoMot+ Z and can be extended to a function GeoMotZ 7→ Z[u, v, (uv)−1 ].

186

BELKALE and BROSNAN

In Z[u, v, (uv)−1 ], there is a distinguished subring, namely, Z[(uv), (uv)−1 ]. Now suppose that all the Kontsevich schemes YG map to this subring. Then one can easily see from Theorem 0.6 (which states that YG ’s generate GeoMot Z ) that the image of GeoMot Z is in this ring. But taking X to be an elliptic curve over Q, and taking a model over Z, we reach a contradiction. Therefore suppose that E(YG ) is not in Z[uv, (uv)−1 ]. There exist p 6= q and a k so that h p,q (Hck (YG , C)) 6= 0. Since Poincar´e duality applies for YG (they are smooth over Q), we can drop compact supports above (and change k to 2 dim(YG ) − k). Therefore we have the following. 15.1 There exists a graph G such that a period of YG is not a multiple zeta value if we grant the following conjecture: If all periods of a cohomology group (of a variety defined over Q) are in the ring of multiple zeta values, then the cohomology is mixed Tate. PROPOSITION

This proposition does not, however, show that Feynman amplitudes coming from graphs need not be (in the Q-span of) multiple zeta values. There are two reasons for this: the precise relation between Feynman amplitudes and the periods of YG has not been worked out yet, and perhaps not all periods of YG correspond to Feynman amplitudes. We will return to the question of the integrals considered by Kontsevich in a later paper [2]. Acknowledgments. We thank T. Chow and N. Fakhruddin for reading several versions of the early stages of this work and B. Totaro for pointing us to Mn¨ev’s universality theorem. In addition, we thank F. Chung, W. Fulton, D. Kreimer, N. Mn¨ev, M. Nori, R. Stanley, J. Stembridge, B. Sturmfels, and C. Yang for useful conversations and email correspondence. The present paper is an updated version of our Max-Planck-Institut f¨ur Mathematik preprint [1]. P. Brosnan thanks the Max-Planck-Institut f¨ur Mathematik in Bonn for providing the wonderful environment in which most of his work on [1] was done. He also thanks his fellow visitors to Bonn, especially, J. Furdyna, R. Joshua, A. Knutson, S. Lekaus, P. Mezo, A. Schwarz, D. Stanley, and B. Toen, for useful and encouraging conversations.

MOTIVES AND A CONJECTURE OF KONTSEVICH

187

References [1]

[2] [3] [4]

[5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19]

[20]

P. BELKALE and P. BROSNAN, Incidence schemes and a conjecture of Kontsevich,

Max-Planck-Institut f¨ur Mathematik, Bonn, Germany, preprint, 2000, http://mpim-bonn.mpg.de/html/preprints/preprints.html 186 , Periods and analytic continuation of integrals, in preparation. 148, 186 J. BOKOWSKI and B. STURMFELS, Computational Synthetic Geometry, Lecture Notes in Math. 1355, Springer, Berlin, 1989. MR 90i:52001 173 D. J. BROADHURST and D. KREIMER, Association of multiple zeta values with positive knots via Feynman diagrams up to 9 loops, Phys. Lett. B 393 (1997), 403 – 412. MR 98g:11101 148 F. CHUNG and C. YANG, On polynomials of spanning trees, Ann. Comb. 4 (2000), 13 – 25. MR 2001g:05041 152, 162 A. CRAW, An introduction to motivic integration, preprint, arXiv:math.AG/9911179 152, 153, 154, 185 J. DENEF and F. LOESER, Germs of arcs on singular algebraic varieties and motivic integration, Invent. Math. 135 (1999), 201 – 232. MR 99k:14002 152, 153, 175 D. EISENBUD, Commutative Algebra with a View toward Algebraic Geometry, Grad. Texts in Math. 150, Springer, New York, 1995. MR 97a:13001 158 W. FULTON, Intersection Theory, Ergeb. Math. Grenzgeb. (3) 2, Springer, Berlin, 1984. MR 85k:14004 157 D. R. GRAYSON, S K 1 of an interesting principal ideal domain, J. Pure Appl. Algebra 20 (1981), 157 – 163. MR 82m:18005 149 ¨ H. GUNZEL , The universal partition theorem for oriented matroids, Discrete Comput. Geom. 15 (1996), 121 – 145. MR 97e:52016 173 M. KONTSEVICH, Gelfand seminar talk, Rutgers Univ., New Brunswick, N.J., December 1997. 148 M. KONTSEVICH and D. ZAGIER, “Periods” in Mathematics Unlimited—2001 and Beyond, Springer, Berlin, 2001, 771 – 808. MR 2002i:11002 154 D. KREIMER, Knots and Feynman Diagrams, Cambridge Lecture Notes Phys. 13, Cambridge Univ. Press, Cambridge, 2000. MR 2002f:81078 148 J. P. S. KUNG, A Source Book in Matroid Theory, Birkh¨auser, Boston, 1986. MR 88e:05028 173 S. MAC LANE, Some interpretations of abstract linear dependence in terms of projective geometry, Amer. J. Math. 58 (1936), 236 – 240. 173 J. MACWILLIAMS, Orthogonal matrices over finite fields, Amer. Math. Monthly 76 (1969), 152 – 164. MR 39:230 159, 163, 183 H. MATSUMURA, Commutative Algebra, Benjamin, New York, 1970. MR 42:1813 ¨ , Varieties of combinatorial types of projective configurations and convex N. E. MNEV polyhedra (in Russian), Dokl. Akad. Nauk SSSR 283 (1985), 1312 – 1314. MR 87f:52010 151, 173 , “The universality theorems on the classification problem of configuration varieties and convex polytopes varieties” in Topology and Geometry: Rohlin Seminar, Lecture Notes in Math. 1346, Springer, Berlin, 1988, 527 – 543.

188

BELKALE and BROSNAN

MR 90a:52013 173 [21]

J. G. OXLEY, Matroid Theory, Oxford Sci. Publ., Oxford Univ. Press, New York, 1992.

[22]

P. W. SHOR, “Stretchability of pseudolines is NP-hard” in Applied Geometry and

MR 94d:05033 169

[23] [24] [25]

[26]

[27] [28] [29] [30]

[31]

Discrete Mathematics, DIMACS Ser. Discrete Math. Theoret. Comput. Sci. 4, Amer. Math. Soc., Providence, 1991, 531 – 554. MR 92g:05065 173 R. P. STANLEY, Spanning trees and a conjecture of Kontsevich, Ann. Combin. 2 (1998), 351 – 363. MR 2001g:05109 149, 151, 152, 154, 156, 161, 162, 167 , Enumerative Combinatorics, Vol. 2, Cambridge Stud. Adv. Math. 62, Cambridge Univ. Press, Cambridge, 1999. MR 2000k:05026 148, 160 J. R. STEMBRIDGE, Counting points on varieties over finite fields related to a conjecture of Kontsevich, Ann. Combin. 2 (1998), 365 – 385. MR 2002a:11062 148, 150, 152, 155 B. STURMFELS, On the matroid stratification of Grassmann varieties, specialization of coordinates, and a problem of N. White, Adv. Math. 75 (1989), 202 – 211. MR 91a:51004 173 I. T. TODOROV, Analytic Properties of Feynman Diagrams in Quantum Field Theory, Internat. Ser. Monogr. Natural Philos. 38, Pergamon, Oxford, 1971. 148 D. J. A. WELSH, Matroid Theory, London Math. Soc. Monogr. 8, Academic Press, London, 1976. MR 55:148 171 H. WHITNEY, On the abstract properties of linear dependence, Amer. J. Math. 57 (1935), 509 – 533. 169 C. YANG, The probabilistic Kirchhoff polynomials over finite fields F p k and a conjecture of Kontsevich, Ph.D. thesis, University of Pennsylvania, Philadelphia, 1999. 152 D. ZAGIER, “Values of zeta functions and their applications” in First European Congress of Mathematics (Paris, 1992), Vol. II, Progr. Math. 120, Birkh¨auser, Basel, 1994, 497 – 512. MR 96k:11110 148

Belkale Department of Mathematics, University of Utah, Salt Lake City, Utah 84112-0090, USA; [email protected] Brosnan Department of Mathematics, University of California, Irvine, Irvine, California 92697, USA; [email protected]

LOW-LYING ZEROS OF DIHEDRAL L-FUNCTIONS E. FOUVRY and H. IWANIEC

Abstract Assuming the grand Riemann hypothesis, we investigate the distribution of the lowlying zeros of the L-functions L(s, ψ), where ψ is a character of the ideal class group √ of the imaginary quadratic field Q( −D) (D squarefree, D > 3, D ≡ 3 (mod 4)). We prove that, in the vicinity of the central point s = 1/2, the average distribution of these zeros (for D −→ ∞) is governed by the symplectic distribution. By averaging over D, we go beyond the natural bound of the support of the Fourier transform of the test function. This problem is naturally linked with the question of counting primes p of the form 4 p = m 2 + Dn 2 , and sieve techniques are applied. Contents 1. Introduction . . . . . . . . . . 2. Explicit formula . . . . . . . . 3. Evaluation of P∞ (φ) . . . . . . 4. Evaluation of P0 (φ) . . . . . . 5. Evaluation of P2 (φ) . . . . . . 6. The density theorem limited . . 7. A partition of P(φ) . . . . . . . 8. Evaluation of P ] (φ) . . . . . . 9. Evaluation of R1 (φ) . . . . . . 10. Preview of results . . . . . . . 11. Estimation of the remainder term References . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

189 196 199 200 201 203 203 204 208 209 210 216

1. Introduction In the recent paper by H. Iwaniec, W. Luo, and P. Sarnak [ILS], the density of zeros near s = 1/2 (i.e., the critical point) of L-functions of families of automorphic forms has been studied thoroughly. The numerous quantitative results of [ILS] give strong DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 2, Received 19 July 2001. Revision received 4 December 2001. 2000 Mathematics Subject Classification. Primary 11M41, 11R42; Secondary 11N36, 11R11. Fouvry’s work supported by the Institute for Advanced Study. Iwaniec’s work supported by National Science Foundation grant number DMS-98-01642, the Ambrose Monell Foundation, and the Hansmann Membership through grants to the Institute for Advanced Study. 189

190

FOUVRY and IWANIEC

evidence for a philosophy and general conjectures proposed by N. Katz and Sarnak [KS1], [KS2]. According to the Katz-Sarnak philosophy, the low-lying zeros of all Lfunctions from a natural family F (when we order them by conductors) are governed by a symmetry group G(F ) associated with F . Lots of relevant symmetries were anticipated and explained in the book [KS1]. More recent results by E. Royer [R] provide further evidence for these general conjectures. This paper builds on [ILS]. We consider the imaginary quadratic field K = √ Q( −D) of discriminant −D ≡ 1 (mod 4), D > 3, and the L-functions for the ideal class group characters ψ : C `(K ) → C∗ , X L(s, ψ) = ψ(a)(N a)−s . (1.1) a

In particular, for the trivial character ψ0 we have the zeta function of K , ζ K (s) = ζ (s)L(s, χ D ),

(1.2)

where χ D is the real character of conductor D (the Kronecker symbol χ D (n) = (−D/n)). The completed functions  √ D s 3(s, ψ) = 0(s)L(s, ψ) (1.3) 2π are entire, except for a simple pole at s = 1 if ψ is the trivial character, in which case √ D 1 res 3(s, ψ0 ) = L(1, χ D ) = h(−D), (1.4) s=1 2π 2 where h(−D) = |C `(K )| is the class number. Moreover, our L-functions are selfdual in the sense of the functional equation 3(s, ψ) = 3(1 − s, ψ).

(1.5)

These properties follow at once from the integral representation due to E. Hecke [H], Z ∞  iy  h(−D) δψ = (y s−1 + y −s ) f ψ √ dy, (1.6) 3(s, ψ) + 2s(1 − s) D 1 where δψ = 1 if ψ is trivial and vanishes otherwise, and X f ψ (z) = ψ(a)e(z N a).

(1.7)

a

If ψ is not real (not a genus character), then f ψ is a cusp form of weight one, level D, and nebentypus χ D ; that is, f ψ ∈ S1 (00 (D), χ D ) and f ψ is primitive (a newform).

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

191

Throughout we assume the Riemann hypothesis for all L(s, ψ). Accordingly, we denote the nontrivial zeros of L(s, ψ) by 1 + iγψ . 2

ρψ =

(1.8)

They appear in complex conjugate pairs by virtue of the functional equation (1.5). Let φ(x) be a smooth function on R which is even and whose Fourier transform Z ˆ φ(y) = φ(x)e(−x y) dx (1.9) R

has compact support, say, supp φˆ ⊂ (−ϑ, ϑ). By Fourier inversion, φ(x) =

Z

(1.10)

∞

ˆ φ(y)e(x y) dy. −∞

Hence φ(x) is of Schwartz class, so it satisfies the bound φ(x)  (1 + |x|)−A

(1.11)

for any A > 0, and φ(x) admits analytic continuation to an entire function of order one. We wish to evaluate the sum  X  γψ D(φ; ψ) = φ log D (1.12) 2π γ ψ

asymptotically as D → ∞. (Here the zeros are counted with corresponding multiplicity.) Certainly this goal is not realistic for any single character ψ because D(φ; ψ) captures an essentially bounded number of zeros of L(s, ψ) in a distance O(1/ log D) to the central point s = 1/2. Therefore we consider the average B (D; φ) =

1 h(−D)

X

D(φ; ψ).

(1.13)

\ ψ∈C `(K )

In other words, we count all the zeros of ζ L (s) =

Y

L(s, ψ),

(1.14)

\ ψ∈C `(K )

which is the zeta function of the Hilbert class field L/K . Our sum B (D; φ) corresponds in [ILS] to the sum Bk∗ (N ; φ) over the zeros of L(s, f ) with f running over primitive cusp forms of weight k (even), level N (squarefree), and trivial nebentypus. In our case, k = 1, N = D, and the nebentypus

192

FOUVRY and IWANIEC

(the central character) is χ D . There are substantial differences between the above cases, which can be noticed instantly. First, the number of forms f being used in Bk∗ (N ; φ) satisfies |Hk∗ (N )| =


k−1 ϕ(N ) + O (k N )2/3 12

(1.15)

(which is quite large), while the number of forms f ψ being used in B (D; φ) satisfies √ √ D  h(−D)  D log log D (1.16) log log D by (1.4) and the Riemann hypothesis for L(s, χ D ). Moreover, J.-P. Serre [S] conjectures that almost the whole space S1 (00 (D), χ D ) (for D prime at any rate) is spanned by f ψ (the dihedral Galois representations), the complementary subspace having dimension bounded by O(D ε ), that is,

1 dim S1 00 (D), χ D = h(−D) − 1 + O(D ε ), 2 if D is prime ≡ 3 (mod 4) (see the unconditional estimates by W. Duke [D]). Therefore our family of L-functions is quite small relative to the conductor. Not only does the smaller family offer a more challenging problem and a closer look at the zeros of an individual L-function, but above all it obeys a distinct law with regard to the distribution of zeros near the central point. We believe in the following. DENSITY CONJECTURE

For any φ that is even and smooth such that φˆ has compact support, Z ∞ B (D; φ) = φ(x)W (Sp)(x) dx + o(1)

(1.17)

−∞

as D → ∞, D ≡ 3 (mod 4), D squarefree, where the density function is given by W (Sp)(x) = 1 −

sin 2π x . 2π x

(1.18)

In contrast, the corresponding density for the family of L-functions for cusp forms of weight k > 2 is shown in [ILS] to be 1 W (O)(x) = 1 + δ0 (x) 2

(1.19)

if |x| < 2, and probably W (O)(x) applies everywhere. Here δ0 (x) denotes the Dirac distribution at x = 0. There is another contrast between the two cases which is relevant for the above distinct densities; namely, the symmetric square L-function associated to any cusp form f ∈ Sk (00 (N )), k > 2, is entire, while in our case the

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

193

symmetric square L-functions associated to dihedral representations have a pole at s = 1 (a double pole if ψ is real, and a simple pole otherwise). As explained in [KS1], the density function W (Sp)(x) is associated with the scaling limit of symplectic groups Sp(2n). By the Plancherel theorem, we have Z ∞ Z ∞ ˆ Wˆ (Sp)(y) dy, φ(x)W (Sp)(x) dx = φ(y) (1.20) −∞

−∞

where Wˆ (Sp) is the Fourier transform of W (Sp) in the sense of distributions; explicitly, 1 Wˆ (Sp)(y) = δ0 (y) − η(y), (1.21) 2 where η(y) is the characteristic function of the segment |y| 6 1, or more appropriately,    1, if |y| < 1, η(y) =

1 2,   0,

if y = ±1,

(1.22)

if |y| > 1.

Our goal in this paper is to prove the density conjecture for test functions φ with the support of φˆ as large as possible. First we verify the conjecture (quite easily) for φˆ ˆ supported in [−1, 1]. In this range the functional (1.20) simplifies to φ(0) − φ(0)/2, so our first result reads as follows. 1.1 Let the conditions be as above. If φˆ is supported in [−1, 1], then  log log D  1 ˆ B (D; φ) = φ(0) − φ(0) + O . 2 log D THEOREM

(1.23)

A little bit stronger asymptotic is established in Theorem 6.1. It is interesting to extend the support of φˆ beyond the segment [−1, 1] because of the discontinuity of Wˆ (Sp)(y) at y = ±1. We are not able to do it with the Riemann hypothesis alone (for the relevant L-functions). By virtue of averaging over the class group characters, the prime numbers of type p=

1 2 (m + Dn 2 ) with n > 0 4

(1.24)

appear, and we need an asymptotic formula for such primes with a reasonably good error term. Euler thought about the primes (1.24) for over forty years, and he discovered a number of their properties, so it is fair to call them Euler primes. Today we know that the Euler primes (precisely, the primes of type p = m 2 + Dn 2 ) are charac√ terized as being those that split completely in the Hilbert class field of K = Q( −D);

194

FOUVRY and IWANIEC

therefore they are prime divisors of an irreducible polynomial of degree h(−4D) (see more details in the wonderful book by D. Cox [C, Th. 9.2]). As beautiful as the arithmetic of the Hilbert class field is, however, it is not an adequate platform for counting the Euler primes with required precision. Not even the Riemann hypothesis for the corresponding zeta function (1.14) is strong enough. One can show that the density conjecture is equivalent to the following. EULER PRIMES CONJECTURE

ˆ Let φ(x) be even and smooth on R with φ(y) of compact support. Then X

φˆ

m 2 +Dn 2 =4 p

 log p  log D

√

log p p log D

 log D  1 1 φ − φ(0) + = h(−D) 4πi 4

∞

Z

φ(x) 0

sin 2π x dx + o(1) (1.25) 2π x

as D → ∞, D ≡ 3 (mod 4), D squarefree. We present heuristic arguments that support the Euler primes conjecture. Moreover, using the Riemann hypothesis for Dirichlet L-functions, we establish the formula (1.25) for almost all odd discriminants −D, provided φˆ has support in (−4/3, 4/3). In particular, we obtain the following. THEOREM 1.2 Suppose that φ is even and smooth such that φˆ is supported in (−ϑ, ϑ) with 0 < ϑ < 4/3. Let 1 > 3, and let D be any set of squarefree numbers D ≡ 3 (mod 4) with 1 < D 6 21 of cardinality |D | > 1ϑ−1/3 . Then Z ∞ 1 X log log 1 φ(x)W (Sp)(x) dx  , (1.26) B (D; φ) − |D | log 1 −∞ D∈D

where the implied constant depends only on ϑ and the test function φ. It turns out that the symplectic density W (Sp)(x) is not perfect. More precise results are given by a density W (x, χ D ) which varies slightly with the discriminant −D (see Prop. 10.1). The refined density W (x, χ D ) has four constituents given by (3.2), (4.2), (5.7), and (9.5). We give asymptotic expansions of these constituents in (3.4), (4.6), (5.13), and (9.9), respectively. From the leading terms of these expansions, we gather that W (x, χ D ) = 1 −

 (log log D)3  σ (χ D ) sin 2π x + (2 sin π x)2 + O |x| , 2π x log D (log D)2

(1.27)

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

195

where σ (χ D ) =

X p

χ D ( p)

log p L0 − log 8π = − (1, χ D ) + O(1). p−1 L

(1.28)

By Proposition 10.1 and Proposition 11.1 (take δ = 1/2 max (0, ϑ −1/3)), we deduce the following. 1.3 Suppose that φ is even and smooth such that φˆ is supported in (−ϑ, ϑ) with 0 < ϑ < 4/3. Then for any 1 > 3 we have Z ∞ 2 X[ φ(x)W (x, χ D ) dx  1β+ε , (1.29) B (D; φ) − THEOREM

−∞

1
where β = max(3ϑ − 1/3, (3ϑ +1)/6), the implied constant depending only on ε and P the function φ. Here and hereafter [ restricts the summation to squarefree integers congruent to 3 (mod 4). By (1.27), we find two extra asymptotic terms for the above functional with respect to the refined measure Z ∞ Z ∞ φ(x)W (x, χ D ) dx = φ(x)W (Sp)(x) dx −∞

−∞

 (log log D)3 
2σ (χ D ) ˆ ˆ + φ(0) − φ(1) +O . log D (log D)2

(1.30)

Note that for special discriminants we may have σ (χ D )  log log D. This happens if χ D ( p) = 1 holds more often than χ D ( p) = −1 for small primes, that is, if K = √ Q( −D) has a lot of small prime ideals of degree one. Since there is a large selection ˆ ˆ of the test functions φ(x) > 0 such that φ(0) > φ(1) (any nonnegative and not identically zero Schwartz function on R is good), our observations suggest that for the special discriminants described above the corresponding dihedral L-functions have a slight surplus of zeros near the central point s = 1/2. Remarks. In retrospect, of course, one could absorb the extra terms on the right-hand side of (1.30) into the main integral by an adjustment in scaling of the zeros; precisely, replace D in (1.12) by D1 = De−2σ (χ D ) . Similarly, one could absorb more terms from the asymptotic expansions (3.6), (4.7), (5.14), and (9.11), but this is only a technical matter.

196

FOUVRY and IWANIEC

2. Explicit formula Let G(s) be a holomorphic function in the vertical strip −1 6 Re s 6 2 such that G(s) = G(1 − s),

(2.1)

s 2 G(s)  1.

(2.2)

By contour integration of G(s)30 (s, ψ)/3(s, ψ), and using the functional equation 30 30 (s, ψ) + (1 − s, ψ) = 0, 3 3

(2.3)

we deduce that X ρψ

1 G(ρψ ) − 2G(1)δψ = 2πi

We have 3(s, ψ) = p = ∞), with

Q

p

Z (2)

2G(s)

30 (s, ψ) ds. 3

(2.4)

L p (s, ψ), where p runs over prime ideals of K (including L p (s, ψ) = 1 − ψ(p)(N p)−s

if p 6= ∞, and L ∞ (s, ψ) =

 √ D s 2π


−1

(2.5)

0(s).

(2.6)

X

(2.7)

Hence (2.4) becomes X

G(ρψ ) − 2G(1)δψ =

ρψ

with Hp (ψ) = −2

Hp (ψ)

p

∞ X

ψ(pν )F(N pν ) log N p

(2.8)

ν=1

if p 6= ∞, and D 2 H∞ (ψ) = F(1) log 2 + 2πi 4π

Z (1/2)

00 (s)G(s) ds. 0

Here F(y) is the inverse Mellin transform of G(s), Z 1 F(y) = G(s)y −s ds. 2πi (1/2)

(2.9)

(2.10)

Equation (2.7) is called the explicit formula (`a la Riemann). We specialize to   1  log D G(s) = φ s − , (2.11) 2 2πi

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

197

where φ is the test function from the introduction. In this case, F(y) =

φˆ (log y/ log D) , √ y log D

(2.12)

and (2.7) becomes D(φ; ψ) − 2δψ φ where Pp (φ; ψ) = −2

∞ X

 log D  4πi

ψ(pν )φˆ

ν=1

=

X

Pp (φ; ψ),

(2.13)

p

 ν log N p  log N p (N p)−ν/2 log D log D

(2.14)

if p 6= ∞, and Z ∞  2 log 2π  2 00  1 2πi x  ˆ P∞ (φ; ψ) = φ(0) 1 − + φ(x) + dx. (2.15) log D log D −∞ 0 2 log D Note that the local terms Pp (φ; ψ) with p = ∞ or deg p = 2 do not depend on the class group character ψ. We write P∞ (φ) in place of P∞ (φ; ψ). Let P2 (φ) denote the contribution to the explicit formula of all primes of degree two. The prime ideals of degree two are principal; p = ( p) with N p = p 2 and χ D ( p) = −1, so X
 2 log q  3(q) P2 (φ) = −2 1 − χ D ( p) φˆ , (2.16) log D q log D (q,D)=1

where p denotes the prime factor of q. All the prime ideals of degree one (ramified or split primes) contribute P1 (φ; ψ) = −2

∞ X X

ψ(pν )φˆ

N p= p ν=1

 ν log p  log p p −ν/2 . log D log D

(2.17)

Therefore (2.13) becomes the following. PROPOSITION 2.1 Let φ be even and smooth such that φˆ has compact support on R. Then for any ψ ∈ C `(K ) we have  log D  D (φ; ψ) − 2φ δψ = P1 (φ; ψ) + P2 (φ) + P∞ (φ). (2.18) 4πi

Averaging (2.18) over the class group characters, we obtain  log D  2 φ = P1 (φ) + P2 (φ) + P∞ (φ), B (D; φ) − h(−D) 4πi

(2.19)

198

FOUVRY and IWANIEC

where X

P1 (φ) = −2 pν

φˆ

principal

 ν log p  log p p −ν/2 log D log D

(2.20)

and p denotes the norm of p (so deg p = 1). Next we express P1 (φ) in terms of rational numbers. First we extract from (2.20) the partial sum over the prime powers that are principal ideals generated by rational integers, say, P0 (φ). Note that if pν = (m) and deg p = 1, then p = p, p|D, and ν is even. Conversely, if p|D and ν is even, then pν is principal rational. Therefore P0 (φ) = −2

∞  XX 2νlog p  −ν log p φˆ p . log D log D

(2.21)

p|D ν=1

We denote by P(φ) the remaining part of (2.20), that is, the partial sum restricted √ by pν = (α) with α not rational. The ring of integers of K = Q( −D) is the free √ Z-module generated by 1 and ω = 1/2(1 + −D), so this ring O = Z + ωZ consists of numbers of type α=

√ 1 (m + n −D), 2

m, n ∈ Z, m ≡ n (mod 2).

(2.22)

Since O has two units ±1, every principal ideal pν = (α) which is not rational has the unique generator (2.22) with n > 0. If deg p = 1, then m is determined by N pν = p ν up to the sign (m = 0 if and only if pν = D). Therefore we have P(φ) = −2

XX m 2 +Dn 2 =4l

φˆ

 log l  3(l) , √ log D l log D

(2.23)

where m, n run over integers, n > 0. (Note that m 2 + Dn 2 6= 4l for l = p ν with χ D ( p) = −1 and n > 0.) Adding (2.21) to (2.23), we obtain by (2.19) the following explicit formula for the average of D(φ; ψ). PROPOSITION 2.2 Let φ be even and smooth such that φˆ has compact support on R. Then

B (D; φ) =

 log D  2 φ + P(φ) + P0 (φ) + P2 (φ) + P∞ (φ), h(−D) 4πi

(2.24)

where P(φ), P0 (φ), P2 (φ), and P∞ (φ) are given by (2.23), (2.21), (2.16), and (2.15), respectively.

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

3. Evaluation of P∞ (φ) First we write (2.15) as the functional Z P∞ (φ) =

199

∞

φ(x)W∞ (x) dx,

(3.1)

−∞

where the density function is given by   2 00  1 2πi x  W∞ (x) = 1 + − log 2π + Re + . log D 0 2 log D

(3.2)

Next we develop an asymptotic expansion for W∞ (x) by applying the power series ∞  X 00  1 + s = −γ − 2 log 2 − (−1)a (2a+1 − 1)ζ (a + 1)s a . (3.3) 0 2 a=1

(Combine (8.363.1), (8.365.2), (8.373.1) of [GR].) Actually, we use the approximation by the partial sum with a < A up to the error term O(|s| A ), where A is a positive integer and the implied constant depends only on A if Re s > 0. We get 2(γ + log 8π) log D  2πi x a X
2 (2a+1 − 1)ζ (a + 1) + O |x| A (log D)−A−1 . − log D log D

W∞ (x) = 1 −

0
(3.4) Hence we derive by Z

∞

φ(x)(2πi x)a dx = φˆ (a) (0)

(3.5)

−∞

the asymptotic expansion  γ + log 8π  ˆ P∞ (φ) = φ(0) 1−2 log D X
a+1 −2 (2 − 1)ζ (a + 1)φˆ (a) (0)(log D)−a−1 + O (log D)−A−1 . 0
(3.6) Note that φˆ (a) (0) = 0 if a is odd. In particular, for A = 2 we get from (3.4) and (3.6) the following approximations:
γ + log 8π + O x 2 (log D)−3 , log D 
γ + log 8π  ˆ P∞ (φ) = φ(0) 1−2 + O (log D)−3 . log D

W∞ (x) = 1 − 2

(3.7) (3.8)

200

FOUVRY and IWANIEC

4. Evaluation of P0 (φ) We can write (2.21) as the functional Z P0 (φ) =

∞

φ(x)W0 (x) dx,

(4.1)

−∞

where the density function is given by W0 (x) =

ζ0  2 4πi x  Re D 1 + . log D ζD log D

(4.2)

Here ζ D (s) denotes the partial Euler product for ζ (s) restricted to primes dividing D. Next we develop an asymptotic expansion for W0 (x). Define the real numbers γa (D) as the coefficients in the power series of ζ D0 (s)/ζ D (s) at s = 1; that is, define γa (D) =

∞
(a+1) (−1)a XX ν a  1 log ζ D (s) s=1 = − (log p)a+1 . a! a! pν

(4.3)

p|D ν=1

In particular, for a = 0 we get γ (D) = −

X log p . p−1

(4.4)

p|D

For any a > 0 we have γa (D)  (log log D)a+1 .

(4.5)

Let A be a positive integer. By the truncated expansion X
ζ D0 (1 + s) = γa (D)s a + O |s| A (log log D) A+1 , ζD 06a
where the implied constant depends only on A if Re s > 0, we get   4πi x a   A+1  X 2 A log log D W0 (x) = γa (D) + O |x| . log D log D log D

(4.6)

06a
Hence we derive the asymptotic expansion    2 a+1 X log log D  A+1 P0 (φ) = φˆ (a) (0)γa (D) +O . log D log D

(4.7)

06a
In particular, for A = 2 we get from (4.6) and (4.7) the following approximations:    log log D 3 2γ (D) + O x2 , (4.8) W0 (x) = log D log D   2γ (D) log log D 3 ˆ P0 (φ) = φ(0) +O . (4.9) log D log D

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

201

5. Evaluation of P2 (φ) For notational simplicity we write χ = χ D . Put Y 1 χ ( p)−1 . U (s) = (s − 1) 1− s p

(5.1)

p-D

We have

(s − 1)ζ (s) , ζ D (s)L(s, χ)Q(s, χ) where Q(s, χ) is given by the infinite product Y χ( p)  1 −χ( p) Q(s, χ) = 1− 1 − , ps ps p U (s) =

(5.2)

(5.3)

which converges absolutely in Re s > 1/2. Hence U (s) has analytic continuation to the half-plane Re s > 1/2 by the Riemann hypothesis for L(s, χ). For Re s > 1 we have X
U0 1 (s) = − 1 − χ( p) q −s 3(q), (5.4) U s−1 (q,D)=1

where p denotes the prime divisor of q. ˆ Now, writing φ(y) as the Fourier transform of φ(x), we derive by (2.16) and (5.4) Z ∞ 1 2 U0  4πi x  P2 (φ) = − φ(0) + φ(x) 1+ dx. (5.5) 2 log D −∞ U log D Hence the first term −(1/2)φ(0) appears as half of the residue of −φ(s)s −1 at s = 0. (Use the fact that φ(s) is even.) We write (5.5) as the functional Z ∞ 1 φ(x)W2 (x) dx, (5.6) P2 (φ) = − φ(0) + 2 −∞ where the density function is given by W2 (x) =

2 U0  4πi x  Re 1+ . log D U log D

(5.7)

Next we develop an asymptotic expansion for W2 (x). Define the real numbers βa (D) as the coefficients in the power series of U 0 (s)/U (s) at s = 1. By differentiating (5.1), we derive βa (D) = γa − γa (D) + γa (χ), (5.8) where γa are the coefficients in the expansion ∞

ζ0 1 X (1 + s) = − + γa s a , ζ s a=0

202

FOUVRY and IWANIEC

γa (D) are given by (4.3), and γa (χ) =

∞ X (−1)a X νa  χ( p) (log p)a+1 . ν a! p p

(5.9)

ν=1

For a = 0 we get β(D) = β0 (D) = γ − γ (D) + γ (χ), where γ (χ) =

X

χ( p)

p

log p . p−1

(5.10)

For any a > 0 we have γa (χ) = −


(a+1) 1 log L(s, χ) s=1 + O(1). a!

Hence, by the Riemann hypothesis for L(s, χ), one can show that γa (χ)  (log log D)a+1 ,

(5.11)

where the implied constant depends only on a. Let A be a positive integer. We have X
U0 (1 + s) = βa (D)s a + O |s| A (log log D) A+1 , U

(5.12)

06a
where the error term is derived by the Riemann hypothesis for L(s, χ) and the implied constant depends only on A if Re s > 0 (see (5.8) – (5.11)). Introducing (5.12) into (5.7), we get   4πi x a  log log D  A+1  X 2 W2 (x) = βa (D) + O |x| A . (5.13) log D log D log D 06a
Next, introducing (5.13) into (5.6), we get    2 a+1 X 1 log log D  A+1 P2 (φ) = − φ(0) + βa (D)φˆ (a) (0) +O . 2 log D log D 06a
(5.14) In particular, for A = 2 we get from (5.13) and (5.14) the following approximations:   3  2β(D) 2 log log D W2 (x) = +O x , (5.15) log D log D   2β(D) log log D 3 1 ˆ P2 (φ) = − φ(0) + φ(0) +O . (5.16) 2 log D log D

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

203

6. The density theorem limited Inserting (3.8), (4.9), and (5.16) into (2.24), we get B (D; φ) =

 log D  2 φ + P(φ) h(−D) 4πi    log log D 3 2σ (χ)  1 ˆ + φ(0) 1 + − φ(0) + O , log D 2 log D

(6.1)

where σ (χ ) = β(D) + γ (D) − γ − log 8 π. Here several terms cancel out, giving σ (χ) =

X

χ( p)

p

log p − log 8 π. p−1

(6.2)

ˆ If φ(y) is supported in |y| 6 log X/log D, then  log D  Z ∞ y/2 ˆ φ = φ(y)D dy  X 1/2 (log D)−A . 4πi −∞ Moreover, by trivial estimation of (2.23), we derive P(φ) 

 X 1/2 D

(log D)−A .

Actually, the sum (2.23) is void if 4X 6 D. Hence and by the lower bound for the class number (1.16), formula (6.1) reduces to  2σ (χ)  1 ˆ B (D; φ) = φ(0) 1+ − φ(0) log D 2
1/2 −1/2 +O X D (log D)−A + (log D)−3 (log log D)3 (6.3) for any A > 0, the implied constant depending on A and the test function φ. This formula is meaningful if X  D(log D) A . In particular, we obtain the following. THEOREM 6.1 If φ is even and smooth such that φˆ is supported in [−1, 1], then    2σ (χ)  1 log log D 3 ˆ B (D; φ) = φ(0) 1+ − φ(0) + O , log D 2 log D

(6.4)

where σ (χ ) is given by (6.2) and the implied constant depends only on the function φ. 7. A partition of P(φ) If φˆ has support beyond [−1, 1], then the sum P(φ) does yield a significant contribution to the distribution of low-lying zeros. Of course, P(φ) must kill 2h(−D)−1 φ(log D/4πi) (the contribution of the pole of ζ K (s) at s = 1), yet it does

204

FOUVRY and IWANIEC

not match this term precisely; there is a secondary term that is bounded but not small. However, extracting the two significant terms for P(φ) is a hard problem. In this section we divide the sum P(φ) into two parts according to a heuristic principle (the idea of an asymptotic sieve in [B]) for general sums over primes. The first part is evaluated in Section 8 precisely enough to give us the two main terms in questions, while the second part is expected to be small (because of cancellation of terms due to the sign change of the involved M¨obius function). We write P(φ) in terms of F rather than φˆ (see (2.12), (2.13)), XX F(`)3(`), (7.1) P(φ) = −2 m 2 +Dn 2 =4`

ˆ where m, n run over integers, n > 0. Recall that φ(y) is even and assumed to be supported in |y| 6 log X/ log D. These conditions assert that F(y) satisfies the symmetry equation y 1/2 F(y) = y −1/2 F(y −1 ) (7.2) and that F(y) is supported in X −1 6 y 6 X . Furthermore, recall that G(s) stands for the Mellin transform of F(y),   Z ∞ 1  log D s−1 G(s) = F(y)y dy = φ s − . (7.3) 2 2πi 0 Since φ(x) is a Schwartz function, we have for t real, 1   t log D  G + it = φ  (1 + |t| log D)−4 . (7.4) 2 2π Fix Y subject to 1 6 Y 6 X . We later choose Y depending on D and X to get optimal results. Now we split 3(`) = 3] (`) + 3[ (`), where X 3] (`) = − µ(d) log d (7.5) d|`,d6Y

and 3[ (`) is the complementary sum. Accordingly, P(φ) = P ] (φ) + P [ (φ), where XX P ] (φ) = −2 F(`)3] (`) (7.6) m 2 +Dn 2 =4`

and P [ (φ) is the complementary sum. 8. Evaluation of P ] (φ) Assuming that the cutoff parameter Y is relatively small, we are able to evaluate (approximately) the partial sum  1 X XX P ] (φ) = 2 µ(d)(log d) F (m 2 + Dn 2 ) (8.1) 4 2 2 d6Y

m +Dn ≡0 (mod 4d)

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

205

by using the Riemann hypothesis for ζ K (s) = ζ (s)L(s, χ D ). Splitting the summation in m into residue classes a (mod 2d), we arrange (8.1) as follows: P ] (φ) = 2

X

∞ X

µ(d)(log d)

X

8n (d, a),

n=1 a (mod 2d) a 2 +Dn 2 ≡0 (mod 4d)

d6Y

where X

8n (d, a) =

F

m≡a (mod 2d)

1 4

 (m 2 + Dn 2 ) .

By the Euler-Maclaurin formula X

f (m) =

∞1

Z

−∞

m≡a (mod q)

we get 1 8n (d, a) = d

Z

∞

q

f (x) +

nx − ao  f 0 (x) dx, q

  1  , (x 2 + Dn 2 ) dx + O √ 4 n D 0 where the error term is derived from the bounds 0 6 {(x − a)/(2d)} < 1 and  ∂ 1 2 F (x + Dn 2 )  |x|(x 2 + Dn 2 )−3/2 . ∂x 4 F

1

The leading term for 8n (d, a) does not depend on a; hence Z ∞  ∞  1 log d X νn (d) F (x 2 + Dn 2 ) dx d 4 0 n=1 d6Y
+ O D −1/2 Y (log X )2 ,

P ] (φ) = 2

X

µ(d)

(8.2)

where νn (d) denotes the number of solutions to a 2 + Dn 2 ≡ 0 (mod 4d) in a (mod 2d). This is a multiplicative function in d such that νn ( p) = 1 +

 −Dn 2  . p

(8.3)

By the Riemann hypothesis for ζ K (s) = ζ (s)L(s, χ D ), we get −

X d6Y

µ(d)νn (d)


log d = v(n) + O Y −1/2 (n DY ) , d

(8.4)

where v(n) is the corresponding complete series. This is also given by the infinite product Y νn ( p)  1 −1 v(n) = 1− 1− . (8.5) p p p

206

FOUVRY and IWANIEC

Inserting (8.4) into (8.2), we deduce that
P ] (φ) = R(φ) + O D −1/2 (Y + Y −1/2 X 1/2 )X  , where R(φ) = −2

∞ X

v(n)

∞

Z

F 0

n=1

1 4

(8.6)

 (x 2 + Dn 2 ) dx.

(8.7)

Writing F(y) as the inverse Mellin transform of G(s) (see (2.10)), we get Z Z ∞ Z ∞   1 2 1 2 s G(s)4 (x 2 + Dn 2 )−s dx ds F (x + Dn ) dx = 4 2πi (2) 0 0 √ Z  4 s−1/2 0(s − 1/2) π = G(s) ds. 2πi (2) 0(s) Dn 2 Hence √ Z  4 s−1/2 0(s − 1/2) −2 π R(φ) = G(s) V (2s − 1) ds, 2πi D 0(s) (2)

(8.8)

where V (s) is given by the Dirichlet series V (s) =

∞ X

v(n)n −s ,

(8.9)

1

which converges absolutely for Re s > 1. By (8.3) and (8.5), we find that v(n) =

X µ(d)χ(d) . ϕ(d)

(8.10)

(d,n)=1

Hence V (s) = ζ (s)

X µ(d)χ(d) d

ϕ(d)ζd (s)

,

where ζd (s) is the local zeta function. This is also given by the infinite product  Y 1 χ( p)  1− s , (8.11) V (s) = ζ (s) 1− p−1 p p which yields analytic continuation of V (s) to Re s > 1/2. As a matter of fact, V (s) has analytic continuation to Re s > −1/2. To see this, we write V (s) = ζ (s)L(s + 1, χ)A(s + 1, χ ), (8.12)

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

207

where A(s, χ) is given by the infinite product A(s, χ ) =

Y p-D

1−

 χ( p) 1 + χ( p) −s p + p − p −2s , p−1 p−1 p−1

(8.13)

which converges absolutely if Re s > 1/2. Remarks. There is a technical (yet worth mentioning) distinction between the present case and the three former ones. Here we consider the Dirichlet series V (s) rather than its logarithmic derivative. The point is that we relax primes of the original summation in the explicit formula by application of the partial von Mangoldt function 3] (`) (the sieve idea). Now we move the integration in (8.8) to the line Re s = 1/2 (the bound (7.4) secures the convergence), getting R(φ) = R0 (φ) + R1/2 (φ) + R1 (φ),

(8.14)

where R0 (φ) comes from the residue at s = 1, which is R0 (φ) = −2π G(1)D −1/2

 V (s)  ζ (s)

s=1

.

(8.15)

Then R1/2 (φ) comes from the half-residue at s = 1/2, which is R1/2 (φ) = −G

1 2

V (0),

(8.16)

and R1 (φ) comes as the principal value of the integral on the line s = 1/2 + it, which is Z  h 4 it i −1 ∞  1 0(it) R1 (φ) = √ G + it Re V (2it) dt. (8.17) 2 D 0(1/2 + it) π −∞ We can compute √ the above residues precisely. First we find by (8.11) that R0 (φ) = −2π G(1)/ DL(1, χ), and by (1.4), this becomes R0 (φ) =

 log D  −2 φ . h(−D) 4πi

(8.18)

Next, by V (0) = ζ (0) = −1/2, we derive R1/2 (φ) =

1 φ(0). 2

(8.19)

208

FOUVRY and IWANIEC

9. Evaluation of R1 (φ) We put  √ 2s+1 1 −1 π2 0(s + 1)0 s + V (2s). (9.1) 2 This is a holomorphic function in Re s > −1/4. Next we replace 0(it) in (8.17) by 0(1 + it)/it getting Z ∞ 
dt t log D  R1 (φ) = φ Im D it K (−it) . (9.2) 2π 2πt −∞ K (s) =

Changing the variable t = 2π x/ log D, we arrive at the functional Z ∞ φ(x)W1 (x) dx R1 (φ) =

(9.3)

−∞

with the density function given by W1 (x) = Im

e(x)  −2πi x  K , 2π x log D

(9.4)

which can also be written as W1 (x) =

 2πi x  cos 2π x  2πi x  sin 2π x Re K − Im K . 2π x log D 2π x log D

(9.5)

Next we develop an asymptotic expansion for W1 (x). Define the real numbers κa (D) as the coefficients in the power series for K (s) at s = 0; that is, define κa (D) = K (a) (0)/a!. We have κ0 (D) = K (0) = 2V (0) = 2ζ (0) = −1 by (9.1) and (8.11). Next we compute κ1 (D) = K 0 (0) =

−K 0 00  1  V0 (0) = −2 log 2 − − 2 (0), K 0 2 V

and by (8.11) we derive (V 0 /V )(0) = (ζ 0 /ζ )(0) − γ (χ), where γ (χ ) is given by (5.10). Finally, putting the values (0 0 / 0)(1) = −γ , (0 0 / 0)(1/2) = −γ − 2 log 2, and (ζ 0 /ζ )(0) = log 2 π, we conclude that κ1 (D) = 2 σ (χ),

(9.6)

where σ (χ ) is given by (6.2). Using the Riemann hypothesis for L(s, χ ), one can show that for any a > 0, κa (D)  (log log D)a , (9.7) where the implied constant depends only on a. Let A be a positive integer. We have X
K (s) = κa (D)s a + O |s| A (|s| + 1)(log log D) A+1 , 06a
(9.8)

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

209

where the error term (for 0 6 Re s 6 1/4) is derived from the Riemann hypothesis for L(s, χ ) (see (8.12)), the implied constant depending only on A. Applying (9.8) to (9.4), we get the asymptotic expansion X sin 2π x e(x)  −2πi x a + κa (D) Im 2π x 2π x log D 16a
W1 (x) = −

(9.9)

For a > 1 we have Z

∞

φ(x)e(x)(2πi x)a−1 dx = φˆ (a−1) (1).

(9.10)

−∞

Now, inserting (9.9) into (9.3), we derive by (9.10) the asymptotic expansion Z ∞ sin 2π x R1 (φ) = − dx φ(x) 2π x −∞ X
− κa (D)φˆ (a−1) (1)(− log D)−a + O (log D)−A (log log D) A+1 . 16a
(9.11) In particular, for A = 2 we get from (9.9) and (9.11) the following approximations:  (log log D)3  sin 2π x 2σ (χ) − cos 2π x + O |x| , 2π x log D (log D)2 Z ∞  (log log D)3  sin 2π x 2σ (χ) ˆ R1 (φ) = − φ(x) dx + φ(1) +O . 2π x log D (log D)2 −∞

W1 (x) = −

(9.12) (9.13)

10. Preview of results To complete the evaluation of B (D; φ), it remains to estimate the sum P [ (φ) which is complementary to P ] (φ) (see (7.6)). We recall that P [ (φ) depends on D, so for clarity from now on we refine this notation by writing P [ (D; φ). Therefore we have P [ (D; φ) =

−2 X X  log `  3[ (`) φˆ √ , log D 2 log D ` 2

(10.1)

m +Dn =4`

where 3[ (`), the residual part of 3(`) so to speak, is X 3[ (`) = − µ(d) log d.

(10.2)

d|`,d>Y

Of course, this depends on the cutoff parameter Y ; nevertheless, we do not display Y in the notation P [ (D; φ) because at the end we choose Y to be a function of D.

210

FOUVRY and IWANIEC

Since Y is relatively large (see (10.4)), we believe that P [ (D; φ) is negligible. Before estimating P [ (D; φ) on average with respect to D (in Sec. 11), we sum up the results obtained so far for individual discriminants. We find that the first term of (5.6) cancels out with R1/2 (φ) (see (8.19)). What is left can be stated as the following. 10.1 ˆ Let φ(x) be even and smooth such that φ(y) has support in [−ϑ, ϑ] with ϑ > 0. Let ϑ 1 6 Y 6 X = D . Then we have Z ∞ B (D; φ) = φ(x)W (x, χ) dx −∞
+ P [ (D; φ) + O D −1/2 (Y + X 1/2 Y −1/2 )X  , (10.3) PROPOSITION

where W (x, χ) = W0 (x)+ W1 (x)+ W2 (x)+ W∞ (x), the implied constant depending on  and the test function φ. (Recall that the densities Wν (x), ν = 0, 1, 2, ∞, are defined by (4.2), (9.4), (5.7), (3.2), resp.) For the error term in (10.3), the optimal cutoff parameter is Y = X 1/3 , giving O(D −1/2 X 1/3+ε ), which is negligible if ϑ < 3/2. However, this choice is not good for the remaining term, P [ (D; φ). We choose Y = Dδ

with 0 < δ <

1 , 2

(10.4)

so the error term in (10.3) is small if ϑ < 1 + δ. 11. Estimation of the remainder term We begin by considering general sums of type XX E(D) = λ` ,

(11.1)

m 2 +Dn 2 =4`

where m, n run over integers, n > 0, and λ` are arbitrary complex numbers for L < ` 6 2L with L > 2. In particular, for λ` =

−2  log `  3[ (`) φˆ √ log D log D `

(11.2)

we get E(D) = P [ (D; φ), but this is not exactly a correct choice because we do not allow λ` in (11.1) to depend on D. However, this is a minor problem that we resolve after completing the work with general sums E(D). Our goal is to estimate the second power moment X[ M (1) = |E(D)|2 . (11.3) 1
LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

211

To this end, some conditions about the numbers λ` are needed. We introduce these gradually as the arguments require. (The crucial assertion is derived from the Riemann hypothesis for Dirichlet L-functions.) We partition E(D) into sums of type XX E f (D) = f (m)λ` , (11.4) m 2 +Dn 2 =4`

where f (m) is a smooth function supported in [M, 2M] such that f ν  M −ν ,

0 6 ν 6 4,

(11.5)

√

with M0 < M 6 2 L, and the short partial sum E 0 (D), which takes the remaining terms not entirely covered by the sums (11.4). The latter is estimated trivially as follows: XX |E 0 (D)| 6 |λ` |. m 2 +Dn 2 =4` |m|<2M0

√ Note that 1 6 n 6 2 L/D. Assuming that |λ` | 6 `−1/2 ,

(11.6)

E 0 (D)  M0 D −1/2 log L .

(11.7)

we get One needs O(log L) sums of type E f (D) together with E 0 (D) to cover exactly E(D). Therefore we conclude by (11.7) that
M (1)  M02 + M f (1) (log L)2 , (11.8) where M f (1) =

X[

|E f (D)|2

(11.9)

1
√ for some f satisfying (11.5) with M0 < M 6 2 L. Now we get by Cauchy’s inequality  L 1/2 X D n

|E f (D)|2 6 2

X

2 f (m)λ`

m 2 +Dn 2 =4`

√ because 1 6 n 6 2 L/D. Hence, letting h = Dn 2 run over all integers, we deduce the inequality  L 1/2 M f (1) 6 2 Sf , (11.10) 1

212

FOUVRY and IWANIEC

where Sf =

2 f (m)λ` .

X X h

(11.11)

m 2 +h=4`

Next we are going to relax the equation m 2 + h = 4` and express S f by character sums to various moduli. First, squaring out, we get XX Sf = f (m 1 ) f (m 2 )λ`1 λ`2 m 21 −m 22 =4(`1 −`2 )

=

XX

f (b + a) f (b − a)λ`1 λ`2

ab=`1 −`2

X

=

 `1 − `2   `1 − `2  f b− λ`1 λ`2 . f b+ b b

XX

M
Now we are going to separate `1 , `2 by applying Mellin’s transform. Before this transformation, we introduce a redundant factor g(l1 /L)g(`2 /L), where g is a smooth function supported in [1/2, 3] such that g(x) = 1 if 1 6 x 6 2. Then we write Z Z  x − y x   y  x − y  f b+ f b− g g = f b (u, v)x iu y iv du dv, b b L L where the kernel function f b (u, v) is determined by Mellin’s inversion. Using (11.5), one derives by partial integration that f b (u, v) satisfies the bound f b (u, v) 

1 |u| −2  |v| −2 1+ 1+ T T T

with T = L M −2 . We obtain Z Z X XX Sf = f b (u, v) b

=

Z Z X

(11.12)

iv λ`1 λ`2 `iu 1 `2 du dv

`1 ≡`2 (mod b)

f b (u, v)b−1

X X a (mod b)

b

`

λ` e

 a`  X  −a`   `iu λ` e `iv du dv. b b `

By the Cauchy-Schwarz inequality and the estimate (11.12), we arrive at Z  |t| −2 Sf  1+ S(t) dt, T where S(t) =

X M
1 b

 a`  2 X X λ` e `it . b

a (mod b)

`

(11.13)

(11.14)

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

213

For the forthcoming estimations we are going to replace the additive characters by the multiplicative ones. First we return to the congruence `1 ≡ `2 (mod b), S(t) =

X

XX

λ`1 λ`2

M
 ` it 1

`2

.

Let c = (b, `1 ) = (b, `2 ) and b = cd, say. After dividing by c, we detect the resulting congruence by the characters χ (mod d), getting S(t) =

X M
1 ϕ(d)

2 X X λc` χ (`)`it .

χ (mod d)

(11.15)

`

From this point on we require a good estimate for every sum X Ac (χ, t) = λc` χ(`)`it

(11.16)

L
which appears in (11.15). We are interested in the numbers λ` = 3[ (`)`−s

(11.17)

for s = 1/2 + iγ , where 3[ (`) is given by (10.2). In this case one can show by an appeal to the Riemann hypothesis for L(s, χ) that   L 1/2 
 δχ Ac (χ, t)  c−1/2 1 + cd L(|t − γ | + 1) (11.18) |t − γ | + 1 Y for any  > 0, the implied constant depending only on . The arguments are standard but not immediate, so we give a sketch of the proof of (11.18). Let ξ(x) be the continuous function on R+ such that ξ(x) = 1 if L +1 ≤ x ≤ 2L, ξ(x) = 0 if x ≤ L or x ≥ 2L + 1, and ξ(x) linear otherwise. Note that the Mellin transform of ξ(x) satisfies Z ∞ L 1+Re v ξˇ (v) = ξ(x)x v−1 dx  . (1 + |v|)(L + |v|) 0 For the proof of (11.18), we can assume that L is an integer. Then we have X Ac (χ, t) = χ(c`)λc` χ(`)`it `

1 = 2πi

Z (1)

ξˇ (v)c−v−s Z (v + s − it) dv,

214

FOUVRY and IWANIEC

where Z (w) =

X X `

 µ(d) log d χ(`)`−w

d>Y d|c`

= L(w, χ)

X δ|c

 µ(δk)(log δk)χ(k)k −w .

X

(k,c)=1, δk>Y

Assuming the Riemann hypothesis for L(s, w), we have L(w, s)  (d|w|)ε , µ(k)χ(k)k −w  (cd|w|K )ε ,

X (k,c)=1, k>K

µ(k)(log k)χ(k)k −w  (cd|w|K )ε ,

X (k,c)=1, k>K

for any ε > 0, Re w = 1/2 + ε, χ (mod d), and K ≥ 1. Moreover, for w = 1 and χ = χ0 , we have X µ(k)k −1  (cd K )ε K −1/2 , (k,cd)=1 k>K

X

µ(k)k −1 log k  (cd K )ε K −1/2 .

(k,cd)=1 k>K

The Dirichlet series Z (w) has analytic continuation to Re w > 1/2, except for a simple pole at w = 1 if χ = χ0 with residue  ϕ(d) X X Z = res Z (w) = µ(δk)(log δk)k −1 s=1 d δ|c

 (cdY )ε



(k,cd)=1 δk>Y c 1/2

Y

.

Moving the integration to the line Re v = ε, we obtain Z 1 Ac (χ , t) = δχ ξˇ (1 − s + it)c−1−it Z + ξˇ (v)c−v−s Z (v + s − it) dv. 2πi (ε) Using the above bounds, one derives (11.18). Applying (11.18) to (11.15), we obtain
S(t)  M + LY −1 (|t − γ | + 1)−2 (|t − γ | + 1) L  . Then by (11.13) we get Sf 

L L + (|s|L) . M Y

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

215

Next (11.10) yields M f (1) 

 L 1/2  L L + (|s|L) . 1 M Y

(11.19)

Finally, introducing (11.19) into (11.8) and choosing M0 = L 1/2 1−1/6 , we arrive at M (1)  L(1−1/3 + L 1/2 Y −1 1−1/2 )(|s|L) .

(11.20)

Now we are ready to derive the following result. 11.1 Let φ be even and smooth such that φˆ is supported in [−ϑ, ϑ] with 0 < ϑ < 4/3. Define P [ (D; φ) by (10.1) and (10.2) with the cutoff parameter Y = D δ for 0 < δ < 1/2. Then for any 1 > 3 we have X[ |P [ (D; φ)|2  11−α+ , (11.21) PROPOSITION

1
where

4  3 α = min − ϑ, δ − (ϑ − 1) , 3 2 the implied constant depending only on  and the test function φ.

(11.22)

Proof Since φˆ is supported in [−ϑ, ϑ], the sum (10.1) runs over ` 6 X = (21)ϑ . Split P [ (D; φ) into partial sums over dyadic segments L < ` 6 2L with 2L 6 X . Then write Z   log `  2 1 γ log D  iγ φˆ = φ ` dγ , log D log D π 2π which separates ` and D. Yet our choice Y = D δ implies that 3[ (`) depends on D; however, this (rather weak dependency) can be resolved by the Mellin transform technique. Another possibility is to choose Y = 1δ so that the problem of separation of variables does not exist. Now apply (11.20), giving (the worst case is 2L = X ) M (1)  (1ϑ−(1/3) + 13ϑ−(1/2)−δ )1 .

Since the number of partial sums in question is O(log 1), this completes the proof of (11.21). Recall that the sum P [ (D; φ) is void if ϑ < 1. The estimate (11.21) is interesting if 0 < δ < 1/2 and ϑ < 1 + (2/3)δ; it implies that for almost all D, P [ (D; φ)  D −α/2 ,

216

FOUVRY and IWANIEC

where α > 0. Moreover, on average (11.21) yields useful estimates for |P [ (D; φ)| with respect to a quite small set of discriminants. Indeed, by Cauchy’s inequality, we get the following. COROLLARY 11.2 Let 1 > 3, let D be a set of squarefree numbers D ≡ 3 (mod 4) with 1 < D 6 21, and let ϑ, δ, α be as above. Then X |P [ (D; φ)|  |D|1/2 1(1−α)/2+ . (11.23) D∈D

Acknowledgments. This paper was written in the fall of 1999 during the special year devoted to “Analytic Number Theory and Automorphic L-Functions” at the Institute for Advanced Study, Princeton. We thank the institute for its generous support and for providing us with excellent working conditions. References [B]

E. BOMBIERI, The asymptotic sieve, Rend. Accad. Naz. XL (5), 1/2 (1975/76),

[C]

D. A. COX, Primes of the Form x 2 + ny 2 : Fermat, Class Field Theory, and Complex

243 – 269. MR 58:10799 204

[D] [GR] [H] [ILS]

[KS1]

[KS2] [R] [S]

Multiplication, Wiley Intersci. Publ., Wiley, New York, 1989. MR 90m:11016 194 W. DUKE, The dimension of the space of cusp forms of weight one, Internat. Math. Res. Notices 1995, 99 – 109. MR 95m:11042 192 I. S. GRADSHTEYN and I. M. RYZHIK, Table of Integrals, Series, and Products, Academic Press, New York, 1965. MR 33:5952 199 E. HECKE, Eine neue Art von Zetafunktionen und ihre Beziehungen zur Verteilung der Primzahlen, I, Math. Z. 1 (1918), 357 – 376. 190 H. IWANIEC, W. LUO, and P. SARNAK, Low lying zeros of families of L-functions, Inst. ´ Hautes Etudes Sci. Publ. Math. 91 (2000), 55 – 131. MR 2002h:11081 189, 190, 191, 192 N. M. KATZ and P. SARNAK, Random Matrices, Frobenius Eigenvalues, and Monodromy, Amer. Math. Soc. Colloq. Publ. 45, Amer. Math. Soc., Providence, 1999. MR 2000b:11070 190, 193 , Zeroes of zeta functions and symmetry, Bull. Amer. Math. Soc. (N.S.) 36 (1999), 1 – 26. MR 2000f:11114 190 E. ROYER, Petits z´eros de fonctions L de formes modulaires, Acta Arith. 99 (2001), 147 – 172. MR 2002g:11063 190 J.-P. SERRE, “Modular forms of weight one and Galois representations” in Algebraic Number Fields: L-Functions and Galois Properties (Durham, England, 1975), Academic Press, London, 1977, 193 – 268. MR 56:8497 192

LOW-LYING ZEROS OF DIHEDRAL L -FUNCTIONS

217

Fouvry D´epartement de Math´ematiques, Universit´e Paris-Sud, Bˆatiment 425 Campus d’Orsay, 91405 Orsay CEDEX, France; [email protected] Iwaniec Department of Mathematics, Rutgers University, New Brunswick, New Jersey 08903, USA; [email protected]

SPECIAL VALUES OF ANTICYCLOTOMIC L-FUNCTIONS V. VATSAL

Abstract The purpose of the paper is to extend and refine earlier results of the author on nonvanishing of the L-functions associated to modular forms in the anticyclotomic tower of conductor p ∞ over an imaginary quadratic field. While the author’s previous work proved that such L-functions are generically nonzero at the center of the critical strip, provided that the sign in the functional equation is +1, the present work includes the case where the sign is −1. In that case, it is shown that the derivatives of the Lfunctions are generically nonzero at the center. It is also shown that when the sign is +1, the algebraic part of the central critical value is nonzero modulo ` for certain `. Applications are given to the mu-invariant of the p-adic L-functions of M. Bertolini and H. Darmon. The main ingredients in the proof are a theorem of M. Ratner, as in the author’s previous work, and a new “Jochnowitz congruence,” in the spirit of Bertolini and Darmon. 1. Introduction The object of this paper is to extend the results and methods of [V2], where it was shown how cases of a conjecture of B. Mazur on the behavior of L-functions in an anticyclotomic Z p -extension could be deduced by studying the distribution of Heegner points associated to definite quaternion algebras. The main result of [V2] showed that when the sign in the functional equation is +1, then the special values of the Lfunctions in question are generically nonzero. In this paper, we propose to study the special values modulo a prime λ of Q, and we can offer three new theorems in this direction. The first result (Th. 1.1) determines the Iwasawa µ-invariant of the p-adic Lfunctions of Bertolini and Darmon. We show that µ is usually zero but not always; when µ 6= 0, we give a precise formula for the value, and we give an interpretation of the positivity in terms of congruences. DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 2, Received 13 March 2001. Revision received 1 November 2001. 2000 Mathematics Subject Classification. Primary 11F67; Secondary 11F33. Author’s work partially supported by Natural Sciences and Engineering Research Council grant number RGPIN 228072-00.

219

220

V. VATSAL

The second result (Th. 1.2) pertains to the case where λ has residue characteristic ` 6= p. In this case, we show that the L-values are typically units but, again, that this is not always the case. For technical reasons, Theorem 1.2 applies only to a restricted class of `, but it seems rather likely that the restrictions can be lifted. Finally, we use the result of Theorem 1.2 to transfer our results from the case where the sign in the functional equation is +1 to the case of sign −1, and to derivatives of L-functions. This is achieved by using congruences and the sign-change phenomenon exploited by Bertolini and Darmon; namely, we prove a “Jochnowitz congruence” that relates the nontriviality of a special value modulo λ to the nontriviality of a classical Heegner point on a modular curve. The result is stated in Theorem 1.4. To describe these results more precisely, let g denote a newform on 00 (N ). Let K denote an imaginary quadratic field of discriminant D such that D is prime to N . Write N = N + · N − , where N + is divisible only by those primes that are split in K , whereas N − is divisible only by primes that are inert. We make the assumption that N − is squarefree and divisible by an odd number of primes. Then the L-function L(g, K , s) has a functional equation with sign +1. Furthermore, if χ is an anticyclotomic character with conductor f prime to N D, then the twisted L-function L(g, χ, s) also has a functional equation with sign +1. We are interested in studying the values L(g, χ, 1) as χ varies over characters of p-power conductor for some fixed prime p. In our previous paper [V2], we showed that L(g, χ , 1) 6= 0 for all but finitely many χ of p-power conductor. In this paper, we study the algebraic part of L(g, χ, 1) modulo a given prime λ of Q. Thus we fix a prime λ of Q with residue characteristic `. Then, following a construction of H. Hida, we define a canonical period can g associated to g by saying that (g, g) can , g = η0 where (g, g) is the Petersson inner product on 00 (N ) and where η0 is Hida’s congruence number associated to g. The precise definition is given in §2.4. Here we remark only that η0 measures congruences modulo λ between g and modular forms of level dividing N on 00 (N ). It is known that the quantity L al (g, χ, 1) =

L(g, χ, 1) can g

(1)

is a λ-adic integer. We want to determine the valuation of L al (g, χ, 1) for χ of conductor p n as n → ∞. To state the results, we need to introduce two further numbers Ccsp and CEis associated to g. We give the precise definitions of these in §2.4, and make some further comments in the discussion following Theorem 1.2. For the moment, we content ourselves with saying that Ccsp measures congruences between g

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

221

and cuspforms that do not occur on the quaternion algebra B = B K , f of discriminant N − . On the other hand, CEis measures congruences between g and the space of Eisenstein series. Then we show that the valuation of L al (g, χ, 1) approaches that of 2 as n → ∞. Ccsp · CEis Now let Hn denote the ring class field of K with conductor p n , and let H∞ = S Hn . Let G ∞ = Gal(H∞ /K ), and let G n = Gal(Hn /K ). Then G ∞ = G 1 × 1∞ , where 1∞ is topologically isomorphic to Z p , and G n = G 1 × 1n , where 1n is cyclic of order p n+δ−1 for some positive integer δ. We caution the reader that the normalization here is slightly different from that in [V2]; the Hn here corresponds to Hn−δ there. The present normalization is more convenient for our purposes here. Given any character χ of G n , we may write χ = χt χw , where χt is a tamely ramified character of G 1 and where χw is a wildly ramified character of 1n . It is convenient to study together all characters χ having the same tame part χt . Let us first consider the case where λ is a prime of residue characteristic p. In this case, there are two very different possibilities, depending on whether a p = a p (g) is a λ-adic unit or not. In the first case, we say that p is ordinary, and in the second we say that p is supersingular. In this paper, we consider only the case where p is an ordinary prime (when considering primes λ such that λ | p). In this ordinary case, our question may be formulated in terms of p-adic L-functions. Let O denote a λadically complete discrete valuation ring (DVR) containing the Fourier coefficients of g, as well as the values of a given tame character χt . Then there exists a p-adic L-function L (g, χt , T ) ∈ O [[T ]] satisfying an interpolation property as follows. For every nontrivial root of unity ζ of p-power order, we have L (g, χt , ζ − 1) =

1 L(g, χ, 1) · Cχ , α 2n can g

where α is the unique unit root of the Hecke polynomial X 2 − a p X + p and where the character χ = χt χw is determined by χw (u) = ζ for some fixed topological generator u of 1∞ . The integer n in the interpolation formula √ nis defined so that the conductor n of χ is p . The number Cχ is given by Cχ = D p . We describe the construction of this p-adic L-function in §5, following a method of Bertolini and Darmon [BD2]. An alternative construction of the p-adic L-function was given by B. Perrin-Riou in [P]. According to the Weierstrass preparation theorem, we may write L (g, χt , T ) = π µ · F(X )·U (X ), where π is a uniformizer in O , F(X ) is a distinguished polynomial, and U (X ) is an invertible power series. The number µ is called the µ-invariant of the p-adic L-function. Clearly, one has
lim ordλ L al (g, χt , ζ − 1) = µ. n→∞

Our first theorem may now be stated as follows.

222

V. VATSAL

THEOREM 1.1 Suppose that p is an ordinary prime for g. Then the µ-invariant is given by the formula 2 µ = ordλ (CEis · Ccsp ).

To the best of our knowledge, Theorem 1.1 provides the first class of examples beyond the classical results of B. Ferrero and L. Washington on cyclotomic fields, and those of L. Schneps and R. Gillard on elliptic curves with complex multiplication, where one can determine the µ-invariant of a broad class of p-adic L-functions. (Hida has recently announced a generalization of the latter to the context of Hecke L-functions of CM fields; see [H2].) Note here that it is not always true that µ = 0. However, when µ 6= 0, then the extra powers of p are accounted for by congruences and by the sign-change phenomenon studied by Bertolini and Darmon (see [BD2] and its various sequels). We discuss this point further below. Now we consider the case of ` 6= p. In this case, the results are not quite as satisfactory. Nevertheless, we can still offer the following. THEOREM 1.2 Suppose that λ has residue characteristic ` 6= p. Suppose that ` splits completely in the field Q(χt ) generated by the values of χt , and suppose that it remains inert in Q(µ p∞ ). Then  L(g, χ, 1)  2 ordλ = ordλ (CEis · Ccsp ) can g

for all but finitely many χ = χt χw . It seems rather likely that one can remove the hypotheses on `, at the cost of introducing some technical complications. As stated, Theorem 1.2 requires, in particular, that the fields Q(χt ) and Q(µ p∞ ) be linearly disjoint. This is the case if and only if the character χt has order prime to p. In this case, Theorem 1.2 holds for a positive proportion of λ. In particular, we see that L(g, χ, 1) 6= 0 for all but finitely many χ , provided that the tame group G 1 has order prime to p. This result was already proven in [V2]. Note that we do not require that p be an ordinary prime here. We point out that the proofs of Theorems 1.1 and 1.2 parallel quite closely the arguments of Ferrero and Washington [FW], [W] in the cyclotomic situation. In fact, it was an attempt to extend the method of Ferrero and Washington that was the motivation for the present work. For a discussion of the analogies, we refer the reader to the introduction of [V2]. It may sometimes be useful to invert the procedure of Theorem 1.2. Starting with a given `, we want to know if there exist characters χ such that L(g, χ, 1)/can g is

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

223

a unit. Then Theorem 1.2 is not so useful. Indeed, if λ is given, then it is rather hard to find primes p such that λ is inert even in Q(µ p ): that infinitely many such p exist is the content of Artin’s primitive root conjecture, which is still an open problem. Nevertheless, one can still resolve the question at hand, under a mild hypothesis on the form g. To state the result, we let ρ denote the residual representation associated to g modulo λ. Thus ρ : Gal(Q/Q) → GL2 (O /λ) is such that Tr(ρ(Frob(q)) = aq for all primes q - N `. We let N denote the Artin conductor for ρ, so that N is the so-called minimal level for ρ, in the sense of Serre’s conjectures. Then we have the following. 1.3 Let the prime ` be given. Suppose that ρ is irreducible, and suppose that there exists some prime q dividing N which divides N precisely to the first power. Then there exist infinitely many quadratic fields K and Hecke characters χ of K such that L(g, χ, 1)/can g is a λ-adic unit. THEOREM

The proof of Theorem 1.3 boils down to finding conditions that force the numbers Ccsp and CEis to be λ-adic units, and to making a convenient choice of p. We give the argument in §4.3. We remark that it applies, in particular, when the level N is squarefree. We want to explain how the numbers Ccsp and CEis may be explained in terms of congruences, starting with CEis since it is rather less surprising. As we have already remarked, CEis measures congruences modulo λ between g and the space of Eisenstein series. In terms of Galois representations, this means that the residual representation ρ associated to g modulo λ is reducible. In this case, it is a well-known phenomenon that the L-values are often nonunits. This seems to have first been observed by Mazur for the unique cuspform on 00 (11) with ` = 5. This observation was further pursued by G. Stevens [St]. On the side of Selmer groups (about which we say nothing here), R. Greenberg [G] has observed that an analogous phenomenon holds. The quantity Ccsp is perhaps more interesting. It turns out that Ccsp is a nonunit if and only if there exists no congruence between g and another form h whose level is not divisible by all the primes in N − . To explain why the existence of such a form h forces the L-values of g to be nonunits, we may argue as follows. Suppose that there is a congruence g ≡ h (mod λ), and suppose that there is some prime q dividing N − such that the level M of h is prime to q. For the purposes of this introduction, we assume that there is a unique such prime q. Under this hypothesis, we see that the number of primes dividing M that are inert in K is even. (This follows from the fact that N − was assumed at the outset to have an odd number of prime

224

V. VATSAL

factors.) It follows then that L(h, χ, s) has a functional equation with sign −1. In particular, the L-functions L(g, χ, s) and L(h, χ, s) have functional equations with opposite signs for any anticyclotomic character χ of conductor p n . Indeed, we find that L(h, χ , 1) vanishes trivially, while L(g, χ, 1) is expected to be nonzero. Since the general philosophy of congruences (see [V1]) suggests that the algebraic parts of L(g, χ , 1) and L(h, χ, 1) should be congruent, we are led to expect that the algebraic part of L(g, χ , 1), being congruent to zero, is a nonunit, and this is precisely what Theorem 1.3 says. The sign-change phenomenon described above was first suggested in [J]. It was further studied and refined by Bertolini and Darmon in the paper [BD2] and its various sequels, especially [BD5]. We exploit a variant of this notion to transfer our results to the case of forms with the functional equation having sign −1. To state the result, let g now denote a form of level N , where all primes dividing N are split in K . (This is the classical Heegner hypothesis.) Then L(g, K , s) has a functional equation with sign −1. If χ is a character of conductor p n with ( p, N D) = 1, then L(g, χ, s) has sign −1 as well. In this case, a conjecture of Mazur (see [M] or [V2]) asserts that L 0 (g, χ , 1) is nonzero for almost all χ . In attempting to prove this conjecture, our only signpost is the Gross-Zagier theorem, which states that if χ0 is the trivial character, then L 0 (g, χ0 , 1) is, up to a simple constant, the height of a classical Heegner point Q on the modular curve X 0 (N ). In particular, if E denotes the abelian variety quotient of J0 (N ) associated to g by Shimura and if π : J0 (N ) → E is the modular parametrization, then the Gross-Zagier theorem states that L 0 (g, χ0 , 1) is nonzero if and only if Q˜ = π(Q) has infinite order in E(K ). On the algebraic side, V. Kolyvagin showed that if Q˜ has infinite order, then E(K ) has rank 1. We are interested in generalizations of this result to twisted Heegner points of higher level. On the algebraic side, the result is due to Bertolini and Darmon. To state their result, let χ denote an anticyclotomic character of conductor c, and let Q ∈ J0 (N ) be the CM point defined by a pair (A, n), where A is an elliptic curve with complex multiplications by the order oc and where n is a fractional ideal of oc with norm N . Then Q is defined over K (c). By abuse of notation, we continue to write Q for the point (Q − ∞) ∈ J0 (N ), where ∞ denotes the cusp at ∞ on X 0 (N ). P Then Bertolini and Darmon [BD1] have shown that if σ χ(σ )Q σ is nonzero in E(K (c)) ⊗ C, then the χ-eigenspace of E(K (c)) has rank 1. Their result is purely algebraic, and it makes no reference to the derivative of the L-series. On the other hand, the Gross-Zagier formula has recently been generalized by S.-W. Zhang to give a formula for L 0 (g, χ, 1) if χ is a ramified character satisfying some very mild conditions (see [Z] for a very general statement, valid for automorphic forms over totally real fields). In particular, Zhang’s generalization holds for charac-

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

225

ters of conductor p n , where p is as above. It follows from his results that L 0 (g, χ , 1) P is nonzero if and only if σ χ(σ )Q σ has nonzero height. P σ In this paper, we study the question of whether σ χ (σ )Q is nonzero in E(K (c)) ⊗ Q(χ) as χ varies over anticyclotomic characters of p-power conductor. In view of Zhang’s results, this may be viewed as a statement about the derivatives L 0 (g, χ , 1). THEOREM 1.4 Suppose that all primes dividing N are split in K . Let χ vary over the set of antiP cyclotomic characters of p-power conductor. Then σ χ(σ )Q σ and L 0 (g, χ , 1) are nonzero in E(Hn ) ⊗ Q(χ) and C, respectively, for all but finitely many χ .

As we have already remarked, Theorem 1.4 is proven by the method of Jochnowitz congruences. Specifically, we introduce another form h of level N q such that g ≡ h (mod λ), where λ is a suitably chosen prime of Q (with characteristic `), and q is a prime such that q - N Dp` which is inert in K . Then g and h have opposite signs in their functional equations. Specifically, g has sign +1, and h has sign −1. We then P prove the Jochnowitz congruence, which implies that σ χ(σ )Q σ is indivisible by λ if L(h, χ , 1) is a λ-adic unit. This latter hypothesis is satisfied for almost all χ in view of Theorem 1.2. Then we deduce Theorem 1.4 from the fact that the torsion subgroup of E(H∞ ) is finite, so that a point that is nonzero modulo λ | ` for ` sufficiently large must have infinite order. Note that we do not require that p be an ordinary prime here. We want to mention that C. Cornut [C] has given a proof of a similar result which works directly with classical Heegner points, and avoids the machinery of Jochnowitz congruences (at least when p is an ordinary prime; if p is supersingular, it does not seem possible to obtain from his method the fact that all but finitely many points are nontrivial). Cornut’s work relies on the results of this paper, especially Lemma 5.13, in an essential way. On the other hand, our analysis of the so-called genus subgroup (see Prop. 5.3) follows fairly closely ideas introduced in his work. We now describe the proof of the Jochnowitz congruence since that part of the argument may perhaps be quite novel. In particular, our method avoids the analysis of component groups which occurs in [BD5]. For simplicity, we assume that g has rational Fourier coefficients, so that it corresponds to an elliptic curve E. We select the prime λ | ` such that E[`] is irreducible as a Galois module. The prime q is chosen to be a Kolyvagin prime, so that q is inert in K , and Frob(q) acts via complex conjugation on E[`]. In particular, we have E[`] = V+ ⊕ V− , where V± is the ±eigenspace for Frob(q), and each V± is one-dimensional over F` . Let us consider the general problem of showing that a Heegner point Q ∈ E(Hn ) is indivisible by `. The basic fact we use is that if Q is a prime of Q over q, then the

226

V. VATSAL

reduction Q of Q modulo Q is a supersingular point defined over Fq 2 . Furthermore, since q splits completely in Hn , any other point in E(Hn ) reduces to a point in E(Fq 2 ). To show that Q is not divisible by ` in E(Fq 2 ), it is therefore enough to exhibit an isogeny E 0 → E, defined over Fq 2 , and with kernel isomorphic to F` , such that there does not exist y ∈ E 0 (Fq 2 ) which projects to Q in E. Such an isogeny may be constructed as follows. Let E ± = E/V± , and let E ± → E denote the dual isogeny. Let W± ⊂ E ± denote the Cartier dual of V± , which is the kernel of the dual isogeny. Note that, under our hypotheses, the group schemes W± and V± become constant, isomorphic to F` , over Fq 2 . Thus, given x ∈ E(Fq 2 ), we may form F± (x) ∈ W± via the mechanism of the Frobenius substitution of geometric class field theory. One could also describe F± (x) more concretely in terms of Kummer theory. Applying this to the reduction of the CM points, we may conclude that Q is indivisible by ` if the element F± (Q) is nonzero in W± . In our applications, we study divisibility of the twisted Heegner points. Namely, P we want to show that σ χ(σ )F± (Q σ ) ∈ W± ⊗ k is nonzero. To analyze this, we view F± as defining a function Z[6] → W± ∼ = F` , where 6 denotes the set of supersingular points on X 0 (N ). The function F± enjoys a certain compatibility with respect to the Hecke operators, coming from its very definition. Furthermore, one has an evident compatibility with the action of Frob(q), which is the nontrivial automorphism of Gal(Fq 2 /Fq ). But there is another way of constructing such a homomorphism Z[6] → F` , this time starting from the modular form h of level N q which is congruent to g. Namely, one views Z[6] as the Picard group M of a Gross curve of level N , associated to the quaternion algebra B of discriminant q, as in Gross’s special value formula (see §2 and [V2]). The form h, being new at q, defines a homomorphism ψ : M → F` , and one easily checks that, with a judicious normalization and choice of sign in F± , one has the same Hecke compatibility. It now follows from a multiplicity-one theorem, due in this case to Mazur, that F± is equal to ψ. Thus we get X X σ χ (σ )F± (Q σ ) ≡ χ (σ )ψ(Q ) (mod λ). σ

σ

The Jochnowitz congruence follows directly from this since Gross’s special value formula states that the right-hand side of the congruence above is a “square root” of the algebraic part of L(h, χ, 1), and our previous results show that the latter is a λ-adic unit for almost all χ. 2. Gross’s special value formula We want to recall the special value formula of B. Gross and the construction of the p-adic L-function. Since this is amply documented in the literature, we will be brief.

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

227

For more details, we refer the reader to [BD2], [BD4], or [BD6]. We use the notation of [V2, §2]. 2.1 Let N = N + · N − as in the introduction, and fix an oriented Eichler order R ⊂ B of level N + . Then we have the isomorphism ˆ × Rˆ × , Cl(B) = B × \ Bˆ × / Rˆ × = B × \ Bˆ × /Q where Cl(B) denotes the set of conjugacy classes of oriented Eichler orders of level N + . Here the second equality follows from the fact that Q has class number one, ˆ × = Q× · Z ˆ × ⊂ Q× · Rˆ × . Let 0 0 = R[1/ p]× , viewed as a subgroup of so that Q ∼ B ⊗ Q× p = GL2 (Q p ). Then strong approximation gives Cl(B) = 0 0 \G/K , where G = PGL2 (Q p ) and where K = PGL2 (Z p ) is a maximal compact subgroup. Let X denote the Gross curve of level N + associated to the quaternion algebra B, and let M = Pic(X ). Thus M is the free Z-module with basis elements given by Cl(B). Then M is a module for the Hecke ring T. Let ψ = ψg : T → Og ⊂ R denote the canonical homomorphism. Let Mg = M ⊗T R,

where the tensor product is taken with respect to the map ψg : T → R. It is known that Mg is an R-vector space of dimension one. Fix an identification Mg ∼ = R or, equivalently, a nonzero element v ∈ Mg . Then we may view ψ as an R-valued function, also denoted by ψ, on M by requiring that ψ(m) · v = m ⊗ 1. Applying ψ to the basis elements [R] ∈ M , we obtain a function ψ 0 : Cl(B) → R. The choice of a different basis element v in the definition has the effect of scaling ψ 0 by a nonzero constant. Since we are interested in studying special values modulo λ, we need to make our special values integral in some canonical way. Thus it is important to specify the function ψ precisely. To do this, we let O denote a λ-adically complete DVR, containing the Fourier coefficients of g. Let M g denote the submodule of M ⊗ O , where T acts via the character ψ. Then the multiplicity-one theorem states that M g is a free O -module of rank 1. Let w denote a generator of M g . Then we fix the isomorphism Mg → R such that the element v = ψ(w) ∈ Mg corresponds to 1 ∈ R. With this fixed normalization, we get a function ψ 0 : Cl(B) → Og , as above. For notational simplicity, we write ψ instead of ψ 0 . Note also that this normalization implies that there exists C ∈ Cl(B) such that ψ(C) is a λ-adic unit. This follows from

228

V. VATSAL

the fact that the element w ∈ M is a linear combination, with integral coefficients, of the basis elements [C] for C ∈ Cl(B). This is useful in the sequel. 2.2 Let o K denote the maximal order of K . We select an orientation on o K , and we regard this choice as fixed. If on ⊂ K denotes the order of conductor p n , we get an induced orientation on on . In this framework, a Heegner point of conductor p n is a pair P = ( f, R), where R is an oriented Eichler order of level N + and where f : K → B is an oriented embedding such that f −1 (R) = on . We identify pairs ( f, R) and ( f 0 , R 0 ) if they are conjugate via the action of B × . Write X n for the set of Heegner points of conductor p n . The basic facts about Heegner points are as follows. (1) Galois action. There is an action of the group Pic(on ) on the set X n . The set X n is a principal homogeneous space for Pic(on ). Hence if en denotes the order of Pic(on ), there are precisely en distinct Heegner points of conductor p n . (2) Tree structure. Each Heegner point of conductor p n has p+1 neighbors, which are Heegner points of conductor pr for suitable r . If n ≥ 1, then precisely p of these neighbors have level p n+1 , while the remaining one has level p n−1 . This unique Heegner point of level p n−1 is called the predecessor of P. (3) Action of T p . The Hecke correspondence T p associates to Pn the formal sum of its p + 1 neighbors. If K n is the kernel of the natural projection Pic(on ) → Pic(on−1 ), then we have the formal identity X Pnσ = T p (Pn−1 ) − Pn−2 σ ∈K n

(4)

for n ≥ 2. Here Pn−1 is the predecessor of Pn , and Pn−2 is the predecessor of Pn−1 . The function ψ. Each Heegner point P = ( f, R) determines a class [P] = [R] ∈ Cl(B). Thus we may view the function ψ above as being defined on the sets X n by putting ψ(P) = ψ([R]).

2.3 Gross’s special value formula may now be stated as follows. Let χ denote a primitive character of Pic(On ) ∼ = Gal(Hn /K ). Then there exists a period  = g,K , depending on K and g, but independent of n, such that

X σ ∈Pic(On )

2 L(g, χ, 1) · Cχ ∈ Q. χ(σ )ψ(P σ ) = 

(2)

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

229

specify the period  more precisely in §2.4. The number Cχ is given by Cχ = √ We K D p n , so that Cχ2 is the Artin conductor of the dihedral representation IndQ (χ ). Note that, by our normalization of ψ, the quantity on the right is actually integral. Computation of the period 2.4 We want to relate the period appearing in the formula (2) above to the canonical integral periods in [H1] and [V1]. Let O denote a λ-adically complete DVR, and let T be any finite, flat, reduced, O -algebra, equipped with a homomorphism π : T → O . Let I = ker(π), and let η denote a generator of the ideal π(Ann(ker(π ))). Thus η is the so-called congruence number for π . Let T0 denote the Hecke ring (over O ) formed in the usual way from modular forms on 00 (N ), and let π : T0 → O denote the canonical homomorphism associated to the modular form g. Let T = T B denote the Hecke algebra formed by the action of Hecke operators on the module M = Pic(X ) as above, where X is the Gross curve of conductor N + and level N − . This time we take for π the homomorphism ψ associated to g. Applying the construction above to T0 and T B , we obtain congruence numbers η0 and η B . Clearly, we have the divisibility η B | η0 . Following Hida, we define a canonical period can g as follows. If (g, g) denotes the Petersson inner product of g with itself, then we put can g = (g, g)/η0 . 2.5 The period  in the special value formula (2) is given by (g, g) = . ηB Thus we have  = can g · C csp , where C csp = η0 /η B . LEMMA

Proof Let χ denote a primitive character of G n for some n. Then let eχ ∈ Pic(X ) ⊗ Q(χ ) P be defined by eχ = σ ∈G n χ(σ )P σ . Let ( · , · ) denote the canonical intersection pairing Pic(X ) × Pic(X ) → Q, extended to a complex pairing that is skew-linear in the second variable, as in [Gr]. Then the special value formula may be equivalently stated as L(g, χ, 1) (eχ , eχ ) = · Cχ . (g, g) The lemma follows from this formula, together with the definition of the function ψ. Remark 2.6 It is clear from Lemma 2.5 that the periods  and can g need not be equal. As we have

230

V. VATSAL

already remarked in the introduction, this may be explained in terms of congruences. Note that the period  in Gross’s formula actually depends on the field K . Remark 2.7 When dealing with questions of congruence, it is often more convenient to work with forms on 01 (N ) rather than 00 (N ). This is the approach taken in [H1], [St], and [V1]. Hida’s canonical integral period from [H1] is, in fact, defined as Hida = g

(g, g)01 (N ) , η1

where we take the inner product and congruence divisor relative to 01 (N ). Furthermore, if the canonical periods ± g of [V1] are defined, then we have − Hida = + g g · g .

We do not concern ourselves with these alternative periods here, and we mention the formulae only to emphasize the various distinctions and to facilitate future comparisons. Finally, we want to explain the other quantity CEis which appears in the statement of our theorems. We let ν denote the largest integer such that the function ψ is congruent to a constant modulo λν , and we put CEis = λ˜ ν , where λ˜ is a λ-adic uniformizer in O . LEMMA 2.8 Let λ denote a prime of Q, and let ν denote the largest positive integer such that the function ψ : Cl(B) → O is constant modulo λν . Then λν divides the numbers aq − q − 1 for primes q with q - N .

Proof Suppose that ψ is congruent to a constant function modulo λr . Then, choosing an x ∈ M such that ψ(x) is a unit, we have ψ(Tq x) = aq ψ(x). Since the Hecke correspondence Tq has degree q +1, we see immediately that aq −q −1 ≡ 0 (mod λr ). Remark 2.9 Thus the number CEis measures congruences between g and the space of Eisenstein series. It is not clear to us whether λν is the exact greatest common divisor of the numbers aq − q − 1. The truth of this statement seems tied to a multiplicity-onetype theorem for Pic(X ) in characteristic `, where λ | `, but, as is well known, such theorems are extremely delicate. At any rate, we have not pursued this question.

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

231

3. Theta elements 3.1 The special value formula in §2 may be conveniently formulated in terms of theta elements in a suitable group algebra of G n , as in [BD2]. Indeed, one might form X θ= ψ(P)σ, σ ∈G n

where P is some fixed Heegner point of conductor p n . Then, if χ is a primitive character of G n , we have |χ(θ)|2 =

L(g, χ, 1) Cχ . 

However, the specializations of the theta element so defined to imprimitive characters χ do not admit any simple description. To get around this difficulty, we must use the notion of regularized Heegner points, as in [BD2]. To recall the definitions, let E p (X ) = X 2 − a p X + p = (X − α)(X − β) denote the Hecke polynomial of g at p. Given a prime λ of Q, we fix a choice of root, say, α, of E p (X ) which is a λ-adic unit. If the residue characteristic of λ is not equal to p, then αβ = p, so that both α and β are λ-adic units. In this case, we make the choice arbitrarily. If λ lies above p, then the unit root α exists only if a p is a λadic unit, which is to say, if p is an ordinary prime for g. In this case, α is uniquely determined. Let λ be a prime of Q, as above. We fix a λ-adically complete and integrally closed ring O = Oλ which contains the Fourier coefficients of g. Given a Heegner point Pn of conductor p n with n ≥ 1, we let Pn−1 denote the predecessor of Pn . Define the regularized Heegner point P˜n by the formal expression 1 1 P˜n = n Pn − n+1 Pn−1 . α α

(3)

Note that the coefficients of P˜n lie in O since α was assumed to be a λ-adic unit. Furthermore, it follows from the properties listed in §2.2 that if n ≥ 2, then we have X (4) ( P˜nσ ) = P˜n−1 . σ ∈K n

This means that the regularized Heegner points are compatible under the norm from Hn to Hn−1 .

232

V. VATSAL

3.2 Now let 30n = O [G n ], where G n = Gal(Hn /K ), and define ψ( P˜n ) = (1/α n )ψ(Pn )− (1/α n+1 )ψ(Pn−1 ). We fix a Heegner point Pn of conductor p n , and we define the theta element θ(Pn ) ∈ 30n as follows: X θ(Pn ) = ψ( P˜nσ ) · σ. (5) σ ∈G n

One checks that if χ is a character of G n , with conductor pr with 1 ≤ r ≤ n, then we have the specialization formula

1 L(g, χ, 1) χ θ(Pn ) χ −1 θ(Pn ) = 2n · Cχ .  α

(6)

Note also that θ depends on the choice of the point Pn . If we choose another point, then we have Pn0 = Pnσ for some σ ∈ G n , so that χ(θ(Pn )) and χ(θ(Pn0 )) differ by a root of unity. Since we are trying to calculate the λ-adic valuation of χ(θ (Pn )), we may choose the point Pn in any convenient fashion. This observation is important later. Pn0 ,

3.3 It is now natural to select a compatible sequence of Heegner points P1 , P2 , . . . , Pn , . . . , where each Pn is the predecessor of Pn+1 , and to form the inverse limit of the corresponding theta elements. For later use, it is convenient to refer to the data of a Heegner point Pn , together with its predecessor Pn−1 , as an edge. This terminology is motivated by the treelike structure of the Heegner points (see [V2], or [BD4], [BD6]). The regularized Heegner point P˜n introduced above may be more properly associated to the edge eEn with origin Pn−1 and terminus Pn . In this case, we define 1 1 ψ(Een ) = ψ( P˜n ) = n ψ(Pn ) − n+1 ψ(Pn−1 ). α α An end is a sequence of oriented edges eE = (Ee0 , eE1 , . . . , eEn , . . . ), where the origin of eEn is the terminus of eEn−1 . We require that each eEn go from a Heegner point of conductor p n−1 to one of conductor p n . Fix a choice of an end eE as above. For each n, let θn = θ (Pn ) ∈ 30n , where Pn is the terminus of the edge eEn . Then one checks that the elements θn are compatible under the natural projection 30n+1 → 30n , so that we may form 20 = 20 (Ee ) = lim θn ∈ 30 = lim 30n . ← − ← −

(7)

Recall that 30n = O [G n ], where G n = G 1 ×1n and 1n is cyclic of order p n+δ−1 . Thus we have 30 = O [G 1 ][[1∞ ]],

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

233

where 1∞ = lim 1n . If χ is a ramified character of Gal(H∞ /K ), primitive of con← − ductor p n , we have the specialization formula χ(20 ) · χ −1 (20 ) =

1 L(g, χ, 1) · Cχ .  α 2n

(8)

Note that we have said nothing about the specialization of χ to the trivial character or to unramified characters. This is a very interesting question, but we do not discuss it here. Here we remark only that the answer is connected to certain predictable degeneracies in p-adic height pairings, and it is computed by using p-adic uniformization of the Shimura curves associated to indefinite quaternion algebras. We refer the reader to [BD4] and [BD6] for the details. As in [V2], it is convenient to express the character χ as χ = χt · χw , where χt is a tamely ramified character of G 1 and where χw is a wild character of 1n , and to consider together all χ with fixed tame part χt . Thus suppose that χt is fixed. Enlarging O if necessary, we have a homomorphism χt : O [G 1 ] → O . This induces a map χt : 30 = O [G 1 ][[1∞ ]] → O [[1∞ ]]. (9) Applying the homomorphism above to the element 20 , we get a new theta element 2 = 2(χt ) ∈ 3 = O [[1∞ ]]. Then 2 satisfies an interpolation property with respect to characters χw of 1∞ . Namely, it interpolates the “square roots” of special values L(g, χt χw , 1) for the fixed choice of χt . 3.4 Now we consider in more detail the case when λ is a prime of residue characteristic p. In this event, O is a finite extension of Z p , and 3 = O [[1∞ ]] is isomorphic to a power series ring Z p [[T ]]. This isomorphism is realized by selecting a topological generator u of 1∞ , and by sending u to 1 + T . In this case, the element 2(χt ) may be identified with a power series L(g, χt , T ) such that if ζ is a nontrivial root of unity of p-power order, then L(g, χt , ζ − 1) · L(g, χt−1 , ζ −1 − 1) =

1 L(g, χ , 1) · Cχ ,  α 2n

(10)

where χ = χt χw , and the character χw is determined by χw (u) = ζ . Here we assume that χw is actually ramified, in order to apply equation (8). The power series L(g, χt , T ) is a (twisted) p-adic L-function for g. By the Weierstrass preparation theorem, we have L(g, χt , T ) = p µ · F(T ) · U (T ),

234

V. VATSAL

where µ is a nonnegative integer, F(T ) is a distinguished polynomial, and U (T ) is an invertible power series. The integer µ is called the µ-invariant of the p-adic Lfunction. Remark 3.5 It is natural to define the p-adic L-function so as to preserve the symmetry between χ and χ −1 . Furthermore, we also arrange that the period in the interpolation formula is the canonical period defined by Hida. Thus we define
L (g, χt , T ) = L (g, χt−1 , T ) = L(g, χt , T ) · L g, χt−1 , (1 + T )−1 − 1 · Ccsp , where the constant Ccsp is defined so that the period  of (2) and the canonical Hida can period can g are related by C csp · g = . However, it is more convenient in practice to work with the power series L(g, χt , T ) since the corresponding theta elements have a relatively simple form. We also remind the reader that this definition of a p-adic L-function depends on the choice of the end eE that we have made above. One easily verifies that the choice of a different end changes the L-function by an invertible power series. We exploit this freedom later. Finally, we point out that if λ has residue characteristic ` 6= p, then there is no λ-adic analytic function interpolating the special values in question. However, it is still convenient to use the theta elements. 4. Preliminary reductions Our goal is to translate our theorems about L-values into statements about the theta elements introduced in §3, with the idea that one can then study the theta elements in terms of distributions of Heegner points. To carry out this program, it is convenient to separate the case of λ - p and λ | p. The case of λ - p Let ν be the largest integer such that the function ψ is congruent to a constant modulo λν , as above. Our goal is to prove the following proposition. PROPOSITION 4.1 Let the tame character χt be given, and let 2 denote the corresponding theta element. Then, for all n  0, there exists a primitive character χw of 1n such that χ (2) satisfies
ordλ χw (2) = ν.

Assuming Proposition 4.1 and keeping in mind the definitions of the various periods, we may easily deduce Theorem 1.2. Indeed, it follows directly from the next corollary.

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

235

COROLLARY 4.2 Suppose that the tame character χt has order prime to p. Suppose also that the prime λ splits completely in the field Q(χt ) obtained by adjoining to Q the values of χt , and suppose that it is inert in the field Q(µ p∞ ) obtained by adjoining all p-power roots of unity. Then  L(g, χ, 1)  Cχ = 2ν ordλ  for all χ = χt χw primitive of conductor p n with n  0.

Proof By virtue of our hypothesis that λ remains inert in Q(µ p∞ ) and splits completely in Q(χt ), we see that the characters χt χw are all conjugate under the action of a decomposition group Dλ . Applying Proposition 4.1, we find a character χw such that χw (2) has λ-adic value equal to ν. Since all the characters χt χw of conductor p n are conjugate under Dλ , we see that χw (2) has valuation ν for all χw , primitive of conductor p n . Arguing similarly with χt−1 and applying the formula
−1
1 L(g, χ, 1) −1 ) , C = χ 2(χ ) · χ 2(χ χ w t w t  α 2n we obtain the statement of the corollary. 4.3 Theorem 1.3 is also a consequence of Proposition 4.1. For the field K , we simply take any quadratic field in which the prime q is inert but all other primes dividing N are split. Since ρ is irreducible, we see that the number CEis is a λ-adic unit (see Lem. 2.8). Furthermore, since q divides the minimal level N , and divides N to precisely the first power, we see that any modular form g 0 on 00 (N ) congruent to g modulo λ is necessarily special at q. In particular, g 0 occurs in the definite quaternion algebra B ramified only at q. It then follows that the number Ccsp is also a λ-adic unit (see Lem. 2.5). Now consider characters of the anticyclotomic Z p -extension of K , where p is prime with ` 6= p. Equivalently, we take χt = 1. (Here p is a prime with ` 6= p; we specify the choice of p below.) Then Proposition 4.1 states that we can find characters χ = χw such that χ(2) is a unit. Now, in view of equation (8), we want to arrange matters so that χ −1 (2) is also a unit. This does not follow directly from Proposition 4.1, but we may instead argue as follows. The main point is to select p in some convenient fashion. Indeed, it suffices to choose p so that the characters χ and χ −1 are conjugate under the action of the decomposition group Dλ . But if χ takes values in µ pr , then the automorphism ζ 7→ ζ −1 of Q(µ pr ) is induced by the element −1 ∈ (Z/ pr Z)× ∼ = Gal(Q(µ pr )/Q). Thus we

236

V. VATSAL

need to choose an odd prime p such that the cyclic group generated by ` in (Z/ pZ)× r −1 has even order 2t for some t. Indeed, in this case, ` p has order 2t in (Z/ pr Z)× , and r −1 we get `t p = −1, as required. Furthermore, we want to ensure that p is relatively prime to N D`. But it is an elementary matter to find such a p: we simply choose p so that ` is a quadratic nonresidue modulo p, and there are infinitely many such p by quadratic reciprocity. Now we want to reduce the proof of Proposition 4.1 to a statement about the distribution of Heegner points. 4.4 As before, we regard the tame character χt and the end eE as given. We want to study the λ-adic valuation of the numbers χ(2), where χ = χt χw , and χw varies in the set Yn of faithful characters of 1n . Thus we consider the formal expression Sn =

1 p n−2

X X

χ(σ ) P˜nσ ,

χw ∈Yn σ ∈G n

so that ψ(Sn ) = (1/ p n−2 ) χw χ(2), where we extend ψ by linearity. Note that the factor 1/ p n−2 is a λ-adic unit. It is enough to show that if n  0, then ordλ (ψ(Sn )) ≤ ν. Indeed, it would then follow that there exists some χ = χt χw such that 0 ≤ ordλ (χ(2)) ≤ ν. On the other hand, the function ψ is congruent to a constant c P P modulo λν by definition of ν. Thus σ ∈G n χ(σ )ψ(P σ ) ≡ c χ (σ ) ≡ 0 (mod λν ). It follows that χ(2) ≡ 0 (mod λν ) for all χ, and we get ordλ (χ(2)) = ν for at least one χ , as required. Thus we must compute the λ-adic valuation of ψ(Sn ). We start with a formal manipulation. Applying [V2, Lem. 2.11], we obtain   X X Sn = χt (τ ) · P˜nτ − p P˜nτ σ . P

τ ∈G 1

σ ∈K n

Applying the norm compatibility of the regularized Heegner points, this reduces to X τ Sn = χt (τ ) · ( P˜nτ − p P˜n−1 ). τ ∈G 1

Finally, inserting the definition of P˜ n , we get   1 n 1 X 1 1 τ o τ τ τ P − P − p P − P Sn = χt (τ ) · αn n α n+1 n−1 α n−1 n−1 α n n−2 τ ∈G 1 n 1  1 X p  τ p τ o = χt (τ ) · n Pnτ − n+1 + n−1 Pn−1 + n Pn−2 . (11) α α α α τ

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

237

Now the number α is a unit. Thus, for a given end eE, define the formal quantity 1  1 1 ξn = ξn (Ee ) = Pn − 2 + p Pn−1 + Pn−2 . (12) α α α Then we have α n−1 Sn =

X

χt (τ ) · ξn (Ee )τ .

τ ∈G 1

P Our task is to bound the absolute value of χt (τ ) · ψ(ξnτ ), where we define ξnτ and ψ(ξnτ ) in the obvious manner. Recall that we may fix the end eE arbitrarily. Now Proposition 4.1 is an immediate consequence of the following result, whose proof is given in §5. 4.5 Let the character χt be fixed. Then, for n  0, there exist ends eE and dE such that X X

χt (τ ) · ψ ξn (Ee τ ) 6= χt (τ ) · ψ ξn (dE τ ) (mod λν+1 ). PROPOSITION

τ ∈G 1

τ ∈G 1

P In particular, at least one of the numbers e τ )) or τ ∈G 1 χt (τ ) · ψ(ξn (E P Eτ τ ∈G 1 χt (τ ) · ψ(ξn (d )) has valuation less than v + 1. The case of λ | p 4.6 This case is similar but easier. Let the tame character χt be fixed, and let 2 denote the corresponding theta element. Then it is clear from the definitions that χ (2) ≡ 0 (mod λν ) for any χ = χt χw . Thus the µ-invariant of L(g, χt , T ) satisfies µ ≥ ν. It P suffices to show therefore that the power series L(g, χt , T ) = cn T n has at least one coefficient cn with valuation at most ν. Equivalently, we want to find the valuations of the coefficients of 2 ∈ O [[1∞ ]]. But now it is enough to consider the theta elements at finite level n. Thus, with the notation of (7) and (9), we write X 2n = χt (θn ) = cn (σ )σ. σ ∈1n

Then our problem is to show that at least one of the numbers cn (σ ) has valuation at most ν. We can give an explicit formula for the cn (σ ) as follows. Let eE denote the end arising in the definition of the theta element, and write P˜n for the regularized Heegner point corresponding to the edge eEn . Then we have X ψ( P˜n ) · σ, θn = σ ∈G n

238

V. VATSAL

which then leads to cn (σ ) =

X

χt (τ ) · ψ( P˜nσ τ ).

(13)

τ ∈G 1

Thus we find that the coefficient cn (1) of the identity element in 1n is given explicitly by X cn (1) = χt (τ ) · ψ( P˜nτ ). τ ∈G 1

Now Theorem 1.1 follows from the next proposition. PROPOSITION 4.7 There exist ends eE and dE with corresponding regularized Heegner points P˜n and Q˜ n such that X X χt (τ )ψ( P˜nτ ) 6= χt (τ )ψ( Q˜ τn ) (mod λν+1 ). τ ∈G 1

τ ∈G 1

The proof of Proposition 4.7 is given in §5. 5. Proof of the main results In this section, we complete the proofs of Propositions 4.5 and 4.7. Using the results and notation of [V2], we reduce everything to a statement about discrete subgroups in SL2 (Q p ). 5.1 As in [V2], we write T for the Bruhat-Tits tree of SL2 (Q p ). The basic facts we use are as follows. (1) The choice of a base point P = ( f, R) of level 1 determines an origin of T , corresponding to the maximal order R p = R ⊗ Z p . (2) Write 0 1 for the subgroup of 0 0 = R[1/ p]× consisting of elements with reduced norm 1. Then 0 1 ⊂ B × ⊂ SL2 (Q p ) is a discrete cocompact subgroup. Let 0 ⊂ 0 1 denote a fixed torsion-free subgroup of finite index, and put G = 0\T . Note that the graph G is bipartite; this causes us some minor inconvenience in the sequel. (3) Each vertex v of T determines a Heegner point P 0 = Pv of level p n for some n depending on v. The point P 0 is represented by a pair ( f, R 0 ), where the embedding f is the same as that for the base point P, and the Eichler order R 0 is in normal form with respect to R, in the sense that R`0 = R` as Eichler orders in B` for ` 6= p. The class of P 0 in Cl(B) is determined as the image of v in 0 0 \T . (Note that there are other Heegner points that are not of this kind, but those do not concern us in this paper.)

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

(4)

(5)

239

∼ If τ ∈ G 1 and if P 0 , v are as above, then there exists τ p ∈ B × p = GL2 (Q p ), 0τ independent of P and v, such that the class of P is that of the vertex v τ = τ p v. The function ψ : Cl(B) → R induces a function G → R. If L 2 denotes the vector space of functions on the vertices of G , then ψ is an eigenfunction for P the operator ∇ defined by ∇(φ)(v) = w∼v φ(w) for φ ∈ L 2 , and the sum is taken over the p + 1 vertices w of G that are adjacent to v. The eigenvalue of ∇ acting on ψ is the Fourier coefficient a p .

The genus subgroup 5.2 The proofs of Propositions 4.5 and 4.7 are based on studying the image of the vectors Q (Pnτ )τ ∈G 1 in the product τ ∈G 1 G , where G ∼ = Cl(B) is the ideal class group of B. Roughly speaking, one would like to say that the vectors (Pnτ )τ ∈G 1 are uniformly distributed as n → ∞. However, this turns out to be false in general, as the image of the Heegner points is constrained by the geometric action of a certain subgroup G 0 of G 1 coming from genus theory (see [C], [V2]). We now proceed to describe this. Let q denote a prime number such that q divides the discriminant D of K . Let q denote the unique prime ideal of K lying above q. Then q2 = (q), and since q is a rational integer, we see that (q) represents the trivial class in each ring class group Pic(On ). (We have assumed at the outset that (D, p) = 1, so q 6= p.) It follows that Frob(q) has order 2 in each Galois group G n , so that Frob(q) lies in the torsion subgroup G 1 of Gal(H∞ /K ). We let G 0 ⊂ G 1 denote the subgroup generated by the elements Frob(q) as q varies over primes dividing D. It is easy to verify (see [V2, §3.8]) that the group G 0 is isomorphic to (Z/2Z)r , where r is the number of distinct primes dividing D. The elements Frob(q), for primes q dividing D, form a basis for G 0 over Z/2Z. Given τ ∈ G 0 , there exists a unique subset Iτ ⊂ {q1 , q2 , . . . , qr } such that Y τ= Frob(q). q∈Iτ

Conversely, any I ⊂ {q1 , q2 , . . . , qr } is of the form Iτ for a unique τ = Q q∈I Frob(q). For τ ∈ G 0 , we may therefore define a squarefree integer d = dτ by saying that Y d = dτ = q. q∈Iτ

The above facts are implicit in [C], but they do not seem to be stated explicitly there.

240

V. VATSAL

Now let χt denote any fixed tame character of G 1 . We define a modified function ψ∗ on Heegner points as follows: ψ∗ (P) =

X

χt (τ )ψ(P τ ).

(14)

τ ∈G 0

Note that χt (τ ) = ±1 for τ ∈ G 0 since G 0 has exponent 2. Thus ψ∗ takes values in the ring generated by the values of ψ. Note also that ψ∗ depends on χt , although we have suppressed this from the notation. Since we are dealing throughout with a fixed character χt , this should not cause any problems. Furthermore, the function ψ∗ takes on only finitely many values since this is already true of ψ. Now observe that if χt is given, then one has the simple identity X X χt (τ )ψ(P τ ) = χt (τ )ψ∗ (P τ ). τ ∈G 1

τ ∈G 1 /G 0

Let C denote a set of coset representatives for G 1 /G 0 . We will see below that the vectors (Pnτ )τ ∈C satisfy good independence properties. Thus we are led to reformulate everything in terms of the function ψ∗ rather than the original ψ. The following proposition verifies that ψ∗ factors through a finite quotient graph of the tree T , in the same manner as ψ, and also satisfies the same basic properties. It is the analogue in our situation of the level raising that occurs in [C]. We also need an analogue of a lemma of Y. Ihara, which is elementary in this context; the argument given here is drawn from [DT, §2]. We let the notation be as in §5.1. PROPOSITION 5.3 The following statements hold. (1) There exists a finite index subgroup 0 D ⊂ 0 and a function ψ D defined on G D = 0 D \T , such that if the vertex v corresponds to the Heegner point P 0 , then we have ψ D ([v]) = ψ∗ (P 0 ), where [v] denotes the class of v in G D . (2) The function ψ D takes values in the ring O . It is nonzero, and if the original ψ is nonconstant modulo λν+1 , then ψ D is also nonconstant modulo λν+1 .

Proof ˆ × Rˆ × , where R is the EichLet ψ denote our original function on Cl(B) ∼ = B × \ Bˆ × /Q ler order corresponding to our fixed Heegner point P = ( f, R) of level 1. From this viewpoint, the relationship with Heegner points is given by ψ(P 0 ) = ψ(x), where x ˆ −1 = Rˆ 0 as oriented Eichler orders, and P 0 = ( f, R 0 ) as usual. is chosen so that x Rx We want to study the modified functions P 0 7→ ψ(P 0τ ) for τ ∈ G 0 and find an adelic

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

241

function ψ D , left-invariant under B × and right-invariant under an open compact subgroup, to represent ψ∗ . So given τ ∈ G 0 , we let τ˜ denote an id`ele of K whose Artin symbol is τ . We may view τ˜ as an element of Bˆ via the embedding f : K → B. Now define a function ψ D on Bˆ × as X ψ D (x) = χt (τd )ψ(x τ˜d ), d

where the sum is taken over squarefree divisors d of D, τd ∈ G 0 is the corresponding element of the Galois group, and τ˜d is the corresponding id`ele, as explained above. Then ψ D is indeed left-invariant under B × and right-invariant under a suitable open compact subgroup. We proceed to clarify its relationship with ψ∗ . So let P 0 = ( f, R 0 ) be a Heegner point corresponding to the vertex v on the tree T . From the definition of the Galois action, we have P 0τ = ( f, R 0τ ), where R 0τ is such that Rˆ 0τ = τ˜ Rˆ 0 τ˜ −1 , so that P 0 7→ ψ(P 0τ ) is represented by ψ(P 0τ ) = ψ(τ˜ x). This means, in particular, that the function ψ D which appeared above does not represent ψ∗ in any obvious manner (since ψ(τ˜ x) 6= ψ(x τ˜ ) for general x). The key is therefore to choose the point x in some intelligent manner, and we accomplish this by raising the level, exactly as in [C]. Given any Heegner point P = ( f, R 0 ), we define a new oriented Eichler order of level N + D by requiring that R 0D ⊗ Z` = R 0 ⊗ Z` if ` - D, and R 0D ⊗ Zq = R 0 (q) = Rq0 ∩ τ˜q Rq0 τ˜q−1 for q | D. The embedding f : on → R 0 restricts to an embedding on → R 0D . To fix orientations, we choose an arbitrary orientation of the maximal order o ⊂ K at each prime q | D. This induces orientations on each order on , and we fix the orientations on R 0D at primes q | D by requiring that the embedding on → R 0D be orientation-preserving. Now, if R D is the Eichler order of level N + D obtained in this way from our base point P = ( f, R) of level 1, then we claim that ψ∗ (P 0 ) = ψ D (x), where x ∈ Bˆ × is chosen so that x Rˆ D x −1 = Rˆ 0D , as oriented Eichler orders of level N + D. To prove this, let τd ∈ G 0 . Then d = q1 q2 · · · qr for distinct primes qi | D, and τ˜d = τ˜q1 · · · τ˜qr . The id`ele τ˜d is trivial away from the primes qi . Now let the base point P be fixed as above. Let P 0 = ( f, R 0 ) be given. Then our choice of x = (x` ) implies that x` R` x`−1 = R`0 if ` is any prime such that ` - D. If ` = q is a prime dividing D, then write T and T 0 for the conjugates of Rq and Rq0 under τ˜q , respectively. Put R(q) = Rq ∩ T , and put R 0 (q) = Rq0 ∩ T 0 , so R(q) and R 0 (q) are local Eichler orders of level q. By definition, we have xq R(q)xq−1 = R 0 (q) . Note that this implies already that conjugation by xq takes the pair of (local) maximal orders (R, T ) to the pair (R 0 , T 0 ). Since we have chosen x so that it preserves the local orientations, we see that xq Rxq−1 = R 0 and xq T xq−1 = T 0 . Now an easy calculation shows that xq−1 τq−1 xq τq normalizes Rq , so that xq−1 τq−1 xq τq ∈ Rq , and we have xq τq Rq = τq xq Rq for each prime q. By multiplicativity, we get xτd Rˆ = τd x Rˆ for any τd ∈ G 0 , which implies

242

V. VATSAL

that ψ(τ˜ x) = ψ(x τ˜ ) for τ ∈ G 0 . It follows that ψ(P 0τ ) = ψ(x τ˜ ) for τ ∈ G 0 . Finally, one finds that the function P 0 7→ ψ∗ (P 0 ) is given by ψ∗ (P 0 ) = ψ D (x), where the point x is chosen as above. We note that it follows from these considerations that the function ψ D is right0 0 invariant under Rˆ × D , and ψ∗ (P ) depends only on the class of R D in the set of conju+ gacy classes of oriented Eichler orders of level N D. We may now complete the proof of the proposition, beginning with the the first assertion. Indeed, if P 0 = ( f, R 0 ) is a Heegner point corresponding to the vertex v, then, by definition, R 0 is in normal form with respect to the base point R, and it is clear that R 0D is in normal form with respect to R D . Thus we may identify R 0D ˆ we deduce from ψ D with the same vertex v. Applying strong approximation in B, a function on T , also denoted by ψ D , such that ψ D is invariant on the left by the group 0 0D = R D [1/ p]× , which is commensurable with 0 0 = R[1/ p]× . Thus we may simply take 0 D = 0 0D ∩ 0 to obtain the first statement of the proposition. As for the second, it suffices to show that the function ψ D on Bˆ × is nonzero and nonconstant modulo λν+1 . We begin with some general observations. Suppose that B is a definite quaternion algebra, and suppose that R ⊂ B is any Eichler order. Suppose that q is a prime number at which B is split, and such that Rq is a maximal compact subring. Fix isomorphisms Bq ∼ = M2 (Qq ) and Rq ∼ = M2 (Zq ). Let τ˜q denote ˆ any element of Rq with reduced norm q, and let R(q) = Rˆ ∩ τ˜q Rˆ τ˜q−1 . If we write ˆ × Rˆ × and Sq = B × \ Bˆ × /Q ˆ × R(q) ˆ × , then there are two projections S = B × \ Bˆ × /Q ˆ ˆ namely, the (degeneracy maps) Sq → S induced by the two inclusions of R(q) → R, −1 ˆ ˆ identity inclusion R(q) ⊂ R and the conjugation x 7→ τ˜q x τ˜q . We write 1∗ and τ˜q∗ to denote the corresponding pullback maps on k-valued functions, where k is any ring. In our application, we have k = O /λν+1 . Since S and Sq are defined as quotients of Bˆ × , we may view functions on S or Sq as functions on Bˆ × via the natural projection maps; note that from this viewpoint we have 1∗ ψ(x) = ψ(x) and τ˜q∗ ψ(x) = ψ(x τ˜q ) for any function ψ on S. Now let ψ1 and ψ2 denote arbitrary nonconstant k-valued functions on the set S. We claim that the functions 1∗ ψ1 and τ˜q∗ ψ2 are linearly independent. This is easy to see. Indeed, suppose that there is a linear dependence relation; then, by scaling the functions ψi , we may assume that 1∗ ψ1 = τ˜q∗ ψ2 . But 1∗ ψ1 = ψ1 (viewed as a function on Bˆ × ) is right-invariant under SL2 (Zq ) ⊂ Rˆ × by definition. Since ψ2 is also right-invariant under SL2 (Zq ), one finds that τ˜q∗ ψ2 is invariant under τ˜q SL2 (Zq )τ˜q−1 . Since 1∗ ψ1 = τ˜q∗ ψ2 , it follows that 1∗ ψ1 and τ˜q∗ ψ2 are invariant under the group generated by SL2 (Zq ) and τ˜q SL2 (Zq )τ˜q−1 . But it is well known that these two subgroups generate all of SL2 (Qq ). (This is a theorem of Ihara; see [S2].) Since all our functions are left-invariant under B × and right-invariant under the product of Q ˆ × , it follows from strong approximation in Bˆ × that they must SL2 (Qq ) · `6=q R`× · Q

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

243

be constant, which is contradictory to our hypothesis. (Note that 1∗ ψ1 and τ˜q∗ ψ2 are constant on Bˆ × if and only if the original ψ1 and ψ2 are so on S.) We can actually sharpen this observation as follows. It is known that the spaces of functions on S and Sq are endowed with an action of Hecke operators T` for all but finitely many primes `. With the notation above, we have T` = 1∗ τ˜`∗ , where 1∗ is the adjoint of 1∗ (see [DT, §2] for a detailed discussion). So suppose further that the functions ψ1 and ψ2 are eigenfunctions for all but finitely many T` , corresponding to the same system of eigenvalues, so that T` ψi = a` ψi , where the eigenvalue a` is independent of i. We also assume that a` 6= ` + 1 for all `. Note that this already implies that the ψi are nonconstant since the eigenvalue of T` acting on the constants is ` + 1. In this situation, we claim that any nontrivial linear combination a · 1∗ ψ1 + b · τ˜q∗ ψ2 is not only nonzero (which was proven above) but also nonconstant. But this is clear: since the Hecke operator T` commutes with τ˜q∗ if q 6= `, one finds that a · 1∗ ψ1 + b · τ˜q∗ ψ2 is a nonzero eigenvector for T` with eigenvalue a` 6= ` + 1, which implies that it is not constant. But it is now immediate that ψ D is nonzero and nonconstant. Indeed, if we fix a prime q | D, then, using multiplicativity, we can rewrite ψ D as ψ D = 1∗ ψ D/q + χt (τq )τ˜q∗ ψ D/q , where ψ D/q =

X

χt (τd )ψ(x τ˜d ),

(d,q)=1

where this time the sum is taken over squarefree divisors d of D such that d is prime to q. The preceding considerations show that ψ D is nonzero and nonconstant, and it is an eigenvector for T` with eigenvalue a` 6= ` + 1 if the same is true for ψ D/q . But we may now argue inductively to reduce to the case of ψ, which is known to satisfy the requisite conditions. Remark 5.4 In the sequel, we work with the quotient graph G D = 0 D \T rather than G . Thus we use the notation [v] and [P] to denote the image of a vertex v, or the class of a Heegner point P, in G D . We also view the function ψ D as being defined on the vertices of G D . Note that if P and Q are Heegner points such that [P] = [Q] in G D , then we have X X ψ D (P) = χt (τ )ψ(P τ ) = χt (τ )ψ(Q τ ) = ψ D (Q) τ ∈G 0

τ ∈G 0

by definition of G D and ψ D . This is useful in the sequel.

244

V. VATSAL

Note also that the graph G D is bipartite. However, it follows from the discussion above that ψ D factors through a nonbipartite quotient, as was the case for ψ. Indeed, we have already seen that ψ D factors through 0 0D \T , where 0 0D = R D [1/ p]× . But by [BD4, Lem. 1.5], the latter contains an element whose determinant has odd valuation. Proofs of Propositions 4.5 and 4.7 5.5 We want to make a good choice for the end PE = (Ee0 , eE1 , . . . , eEn , . . . ) as in the definition of the theta elements and p-adic L-function. If τ ∈ G 1 , then write eEnτ for the conjugate of en under the action of τ . Let Pn be the origin of the edge eEn . Then Pn is also the terminus of the preceding edge eEn−1 . Since the formula (12) involves the predecessors of Pn , we are motivated to consider the vertices Pn−1 and Pn−2 traversed by the end eE at steps n − 1 and n − 2. Observe that for each n and τ , the vertices [Piτ ] τ ] and of G D corresponding to the Heegner points Piτ satisfy the condition that [Pn+1 τ ] are distinct neighbors of [P τ ]. [Pn−1 n With this notation, we have the following important proposition. It is the analogue in our situation of the main lemma in Ferrero and Washington [FW]. Recall that the graph G is bipartite, so that the vertices of G D are divided into two sets, depending on whether the distance from some given vertex is even or odd. 5.6 Let Q r be a given Heegner point of conductor pr with predecessors Q r −1 and Q r −2 of conductors pr −1 and pr −2 , respectively. Let C be a fixed set of representatives of G 1 /G 0 . Then, for all n ∈ Z sufficiently large with n ≡ r (mod 2), we may find an end eE, and τ0 ∈ C, satisfying the following two conditions: τ ] = [Q τ ] for τ ∈ C, τ 6= τ , and 0 ≤ i ≤ 2; (1) [Pn−i 0 r −i τ τ0 τ0 (2) [Pn 0 ] = [vn ], [Pn−1 ] = vn−1 , and [Pn+1 ] = vn+1 , where the vi are any given vertices of G D such that vn−1 and vn+1 are distinct neighbors of vn , subject to the constraint that [vn ] and [Q r ] are in the same class of the bipartition of GD . The element τ0 ∈ G 1 depends only on the quadratic field K . PROPOSITION

Remark 5.7 We may view the end eE as determining an infinite walk, without backtracking, on the finite graph G D . In concrete terms, Proposition 5.6 means that we can choose eE so that when τ ∈ C, τ 6= τ0 , then the vertices traversed by eE τ at stages n, n − 1, n − 2 are given by the original [Q rτ ], [Q rτ−1 ], [Q rτ−2 ], respectively, while still retaining the freedom to specify the vertices traversed (at the same three stages) by eE τ0 .

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

245

To complement Proposition 5.6, we need to know that we can make good choices for the vertices traversed by eE τ0 . That such choices exist is the content of the following. LEMMA 5.8 There exists a vertex x of G D , together with distinct neighbors y and z, such that ψ D (y) and ψ D (z) are not congruent modulo λν+1 . We may choose the vertex x to lie in either class of the bipartition of G .

Proof We have seen in Proposition 5.3 that the function ψ D on G D is not congruent to a constant function modulo λν+1 . So ψ D takes on at least two distinct values modulo λν+1 . Suppose that, for every vertex x of G D , the function ψ D is constant modulo λν+1 on the neighbors y of x. Since this is true for every x, it would follow that ψ D takes on precisely two values modulo λν+1 , say, a and b. Dividing the vertices of G D into corresponding sets A and B, it follows that every edge of G D goes from A to B. Then G D must be a bipartite graph, and the function ψ D must respect the bipartition. But ψ D factors through the quotient 0 0D \G D of G D , and we have remarked above that 0 0D contains an element that interchanges the two halves of the bipartition. This implies that ψ D is constant modulo λν+1 , a contradiction. Thus we obtain at least one vertex x satisfying our requirements. The fact that we may choose x to lie in either class of the bipartition also follows from the fact that 0 0D contains an element that interchanges the vertices in the bipartition. 5.9 Before embarking upon the proof of Proposition 5.6, we want to show how it implies Propositions 4.5 and 4.7. Let x, y, z denote the vertices of G D provided by Lemma 5.8. Note that the vertex x may be chosen from either half of the bipartition; we have to keep track of this choice in the sequel. Now select a vertex w such that w is adjacent to x but such that w is distinct from both y and z. Such a w exists because G D is regular of degree p+1 ≥ 4. Since w and y are adjacent to x, it is clear from the considerations of [V2] that there exists a Heegner point Z r of level pr , with predecessors Z r −1 and Z r −2 , such that [Z r −1 ] = x, while [Z r +1 ] = y, and [Z r −2 ] = w. Observe that the choice of x determines the parity of r . Now Proposition 5.6 states that, for all n  0 of suitable parity, there exists an end eE (perhaps depending on n), with corresponding Heegner points P j , such that τ ] = [Z τ ] for each i, and τ ∈ C. In particular, the vertices [P τ0 ] traversed [Pn−i r −i j by eE τ0 at stages n − 2, n − 1, n are w, x, y, respectively. Applying Proposition 5.6 E with corresponding Heegner points Q j , such that again, we now find another end d, τ τ [Q n−i ] = [Z r −i ] for i = 0, 1, 2, and τ ∈ C, τ 6= τ0 . At τ = τ0 , we require that

246

V. VATSAL τ

τ

τ

0 0 [Q n−2 ], [Q n−1 ], [Q n0 ] be w, x, z, respectively. Now we compare the associated theta elements. We begin by considering the situation of Proposition 4.5. Fixing some end of T , and letting ξn denote the quantity defined in (12), our job is to compute the valuation of the expression X X χt (τ )ψ(ξnτ ) = χt (τ )ψ D (ξnτ ),

τ ∈G 1

τ ∈C

where ψ D (ξnτ ) = τ ∈G 0 χt (τ )ψ(ξnτ ). But since the function ψ D factors through the finite graph G D , we find that ψ D (ξn (Ee )τ ) = ψ D (ξn (dE )τ ) for τ ∈ C, τ 6= τ0 . On τ0 τ0 τ0 τ0 the other hand, we have [Pn−1 ] = [Q n−1 ] = x and [Pn−2 ] = [Q n−2 ] = w, while τ0 τ0 [Pn ] = y and [Q n ] = z. By the choice of the vertices y and z, and formula (12), we find that P




1 ψ D ξn (Ee )τ0 − ψ D ξn (dE )τ0 = 2 ψ D (z) − ψ D (y) 6= 0 α

(mod λν+1 ).

This gives the statement of Proposition 4.5 for all n  0 of suitable parity. To complete the proof, one simply repeats the argument, upon choosing x appropriately. Finally, one obtains Proposition 4.7 in precisely the same fashion. Proof of Proposition 5.6 5.10 Let P denote a Heegner point, corresponding to the vertex v ∈ T . We want to control the classes in G D of the various P τ for τ ∈ C and then say something about the various predecessors. Then it is natural to introduce the vector (τ p v)τ ∈C , and to study its distribution in the product G D × · · · × G D , where the product is indexed by the elements of C. Note also that G D = 0 D \T . If we put 0 τD = τ p 0 D τ p−1 , and G Dτ = 0 τD \G , then there is an isomorphism G D ∼ = G Dτ induced by v 7→ τ p v. It is convenient in the sequel to reformulate the problem so that we consider the distribution Q of the diagonal vector (v, . . . , v) in τ ∈C G Dτ . Letting K˜ denote a maximal compact ˜ K˜ and subgroup of G˜ = PGL2 (Q p ), we have T = G/ Y Y ˜ K˜ . G Dτ = 0 τD \G/ τ ∈C

τ ∈C

Q We are therefore led to consider 0C \G˜ C , where 0C = τ ∈C 0 τD and G˜ C = ˜ τ ∈C G. In order to do this, we need some simple preliminary results. Note that PSL2 (Q p ) ⊂ PGL2 (Q p ) is a subgroup of index two; it actually is essential to work with the smaller group, as it is a simple group and generated by unipotent elements. Q

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

247

LEMMA 5.11 Let G denote the group PSL2 (Q p ) = SL2 (Q p )/ ± 1. Then G admits no nontrivial automorphism commuting with all inner automorphisms.

Proof Let φ denote any automorphism of G which commutes with all conjugations. Let x and g be arbitrary elements of G; then we have gφ(x)g −1 = φ(gxg −1 ) ⇐⇒ gφ(x)g −1 = φ(g)φ(x)φ(g)−1 since φ commutes with the conjugation by g. This implies that g −1 φ(g) commutes with φ(x), for any x and g. Letting x vary, we find that g −1 φ(g) is in the center of G, so that g −1 φ(g) = 1. LEMMA 5.12 Q Let r ≥ 2 denote an integer, and let G ∗ = ri=1 G, where G = PSL2 (Q p ). Let 1 ⊂ G ∗ denote the diagonal subgroup consisting of the elements 1(x) = (x, . . . , x) for x ∈ G. Let H ⊂ G ∗ denote any nontrivial subgroup that is normalized by 1. Then one of two possibilities occurs: either (1) H = 1 or (2) there is a proper subset S ⊂ {1, 2, . . . , r } such that H has nontrivial intersecQ tion with G ∗S = i∈S G; in this case, H ∩ G ∗S is normalized by the diagonal subgroup 1 S ⊂ G S ; here 1 S consists of the elements 1 S (x) for x ∈ G, where 1 S (x) = (x1 , . . . , xr ) has component xi = x for i ∈ S, and xi = 1 if not.

Proof We note at the outset that H ∩ G ∗S is normalized by 1 S because H is normalized by 1. Now, since H is nontrivial, we may assume, by relabeling if necessary, that the projection of H to the first factor is nontrivial. Then, since H is normalized by 1 and since G = PSL2 (Q p ) is a simple group, we find that the projection of H onto the first factor is surjective. Thus, for any x ∈ G, we may select some φ(x) = (φ1 (x), . . . , φr (x)) ∈ H such that φ1 (x) = x. Suppose that there exists some x ∈ G such that φ(x) is not unique. In this case, there exist distinct elements φ(x), φ 0 (x) in H such that both φ(x) and φ 0 (x) have first component x. Considering φ(x)φ 0 (x)−1 , we find that H has nontrivial intersection with the subgroup G ∗S , where S = {2, 3, . . . , r }. Thus case 2 of the lemma holds. We may therefore assume that, for each x ∈ G, there is a unique φ(x) with first component x. Letting x, y ∈ G, we find that


φ(x)φ(y) = x, . . . , φr (x) · y, . . . , φr (y) = x y, . . . , φr (x)φr (y) .

248

V. VATSAL

Thus φ(x)φ(y) is an element of H with first component x y. By uniqueness, we find that φ(x)φ(y) = φ(x y), since φ(x)φ(y) is an element with first component x y. It follows from the formula above that φi (x)φi (y) = φi (x y) for every x, y, so that each φi is a homomorphism. If φi (x) = 1 for some x and i, we see that H ∩ G ∗S is nonempty for S = {1, 2, . . . , i − 1, i + 1, . . . , r } and we are in case 2. Thus we may assume that each φi is injective. Using the fact that H is normalized by the diagonal, together with the uniqueness, we see, in fact, that each φi : G → G is surjective and commutes with all conjugations. But now Lemma 5.11 implies that each φi is the identity map, so that H is the diagonal subgroup, as stated in case 1. 5.13 Let G ∗ be as in Lemma 5.12. For each i = 1, . . . , r , let 0i denote a discrete and Q cocompact subgroup of G, and let 0∗ = ri=1 0i ⊂ G ∗ . Let X denote the closure of the product 0∗ · 1, where 1 is the diagonal, as above. Suppose that, for i 6= j, the groups 0i and 0 j are not commensurable. Then X contains a subgroup of the form 1 × · · · × G × 1, concentrated on the ith factor for some i with 1 ≤ i ≤ r . LEMMA

Proof As in [V2], the main ingredient is Ratner’s theorem (see [R, Th. 2], [V2, Th. 4.7]) on closures of unipotent orbits. Indeed, Ratner’s theorem implies that the set X is of the form 0∗ · H , where H is a closed subgroup of G ∗ containing 1, such that 0∗ · H is closed in G ∗ . Note that H must strictly contain the diagonal since the groups 0i and 0 j are not commensurable for i 6= j. (This follows from the case when r = 2, which has already been proven in [V2, Cor. 4.8].) Also, since 1 ⊂ H , it is trivial that H is normalized by 1. Choose some (x1 , x2 , . . . , xr ) ∈ H with, say, x1 6= x2 . Since (x1 , x1 , . . . , x1 ) ∈ H , we see that H S = H ∩ G ∗S is nontrivial, where S = {2, 3, . . . , r }. Furthermore, H S is normalized by 1 S . It follows from 0 Lemma 5.12 that H S is either diagonal or has nontrivial intersection with some G ∗S , where S 0 ( S. If H S is diagonal, then we are done by induction on r . If not, we repeat 0 the argument with S 0 instead of S. If H S is diagonal, then we are done, otherwise we can shrink S 0 . Repeating this argument, we reduce to the case when S 0 has just one 0 0 element, and H S = H ∩ G ∗S is nontrivial and concentrated on the ith factor. Since 0 H S is normalized by the diagonal, and PSL2 (Q p ) is simple, we find that H contains the ith factor, as required. 5.14 We may now prove Proposition 5.6. We are interested in triples v = (vn−1 , vn , vn+1 ), where each vi is a vertex of T and where there are oriented edges leading from vn−1 to vn and then from vn to vn+1 . Equivalently, the vertices vn−1 and vn+1 are distinct

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

249

neighbors of vn . Given such a triple v, and an element τ ∈ G 1 , we may define the conjugate vτ in the obvious manner. 0 0 Now let X denote the set of such triples v0 = (vn−1 , vn0 , vn+1 ), where this time 0 the vi are vertices of the finite graph G D , subject to the same adjacency conditions. Then X is a finite set, and we are interested in studying the image of (vτ )τ ∈C inside Q ˜ τ X . So write X for the set of triples v = (vn−1 , vn , vn+1 ) as above, but this time with the vi ∈ T . Then there is an action of G˜ = PGL2 (Q p ) on X˜ by left translation, and, using the description of vertices of T in terms of homethety classes of lattices (see [S2]), one easily checks that this action is transitive. Furthermore, the stabilizer of any given v is an open compact subgroup N (a subgroup of “level” p 2 ). One deduces ˜ from this that X = 0 D \G/N , and that the image of v in X is computed simply as the image of v in the quotient. Furthermore, the class of vτ is simply computed as the image of τ p v. Equivalently, the class of vτ is determined by the image of v in X τ = 0 τD \G/N . Q We apply the foregoing results, taking as the 0i the groups 0 τD . Note that 0 τD ⊂ Q SL2 (Q p ) by definition of the groups 0 τD . Now the points Q i of Proposition 5.6 ˜ determine a triple v, and we let x ∈ G˜ represent the corresponding point in G/N . In view of Lemma 5.13, there exist τ0 ∈ C and y ∈ G = SL2 (Q p ), together with elements γτ ∈ 0 τD for τ ∈ C such that γτ y is very close to 1 in PSL2 for all τ 6= τ0 , and γτ0 y may be specified to lie in an arbitrary open set (again in PSL2 ). Taking the element yx ∈ G, we find that yx determines the same class as x in X τ for τ 6= τ0 , τ while the image of yx ∈ G D0 may be specified arbitrarily, subject to the stated parity condition. Note also that the element τ0 depends only on the quadratic field K . This proves that the desired patterns occur for Heegner points of some large conductor; that the patterns occur for points of conductor p n for all n  0 with n ≡ r (mod 2) follows from Ratner’s theorem on uniform distribution (see [V2, Th. 4.11] or [R, Th. 4]). We want to point out a simple consequence of Lemma 5.13. 5.15 Let τ 7→ Cτ be any function from C to Cl(B). Then there exists a Heegner point P of conductor p n for all n  0 such that [P τ ] = Cτ for each τ ∈ C. PROPOSITION

Proof It follows from Lemma 5.13 and induction on r that 0 D · 1 is dense in G ∗ .

250

V. VATSAL

6. Jochnowitz congruences In this section, we show how the foregoing results may be used to study the nontriviality of classical Heegner points on modular curves. Thus let g denote a modular form of level N , and let K denote a quadratic field in which all primes dividing N are split, so that L(g, K , s) has a functional equation with sign −1. (This is the classical Heegner hypothesis.) Let E denote the abelian variety attached by Shimura to g. Then we S are interested in studying E(H∞ ), where H∞ = Hn is the compositum of all ring class fields of conductor p n , and the goal is to find points of infinite order. As for the torsion points, the following lemma is well known (see [BD2, Lem. 6.3]). 6.1 We have E(Hn ) = Fn ⊕ E n , where E n is Z-free and where Fn is finite of order bounded independent of n. Thus the torsion subgroup F∞ of E(H∞ ) is finite, and Fn = Fm = F∞ for n and m sufficiently large. LEMMA

The basic mechanism for producing points is that of complex multiplication. As in the introduction, we let Q ∈ X 0 (N ) be the CM point defined by a pair (A, n), where A is an elliptic curve with complex multiplications by the order Oc of conductor c, and where n is a fractional ideal of Oc with norm N . Then the point Q is defined over K (c). By abuse of notation, we continue to write Q for the point (Q − ∞) ∈ J0 (N ), where ∞ denotes the cusp at ∞ on X 0 (N ). Let Q˜ denote the image of Q in E(K (c)). We want some criterion for determining whether or not the point Q˜ and its various twists are of infinite order. This is provided by the following simple observation, at least when c = p n . LEMMA 6.2 Let c = p n , and let the groups E n and Fn be as in Lemma 6.1. Let ` denote a prime S number such that the order of F∞ = Fn is prime to `. Let Q˜ be any point in E(Hn ). Then Q˜ has infinite order if and only if it is nonzero in E n ⊗ F` . More generally, let χ be a character of Gal(Hn /K ), and let λ | ` denote a prime P of Z[χ] above `. Write k for the residue field of Z[χ] at λ. Then Q˜ χ = χ(σ ) Q˜ σ has nonzero height if and only if Q˜ χ is nonzero in E(Hn ) ⊗ k.

Proof The proof is clear because Fn ⊗ F` = 0 by our choice of `. Note also that the height pairing is nondegenerate on the free part of E(Hn ), and is isotropic on the torsion part Fn .

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

251

Preliminary choices We prove our main result (Th. 1.4) by relating the nonvanishing of Q˜ χ in E(Hn ) ⊗ k to the nonvanishing modulo λ of the special value L(h, χ, 1), where h is a suitably chosen form of level N q, congruent modulo λ to the original g, and by applying our previous results to show that the latter is a unit for almost all χ . We begin by specifying more precisely the choices of ` and q which intervene in this program. It is convenient in the sequel to write χ = χt χw and to group together all characters with a fixed tame part χt . To make the choices below, we assume that the tame character χt is fixed. 6.3 P Let the modular form g = an (g)q n be as above. Then, for any prime λ of Q, of residue characteristic `, there exists a Galois representation ρ = ρλ : Gal(Q/Q) → GL2 (O ), where O is a λ-adically complete DVR such that Tr(Frob(q)) = aq (g) ∈ O for q - N `. If k denotes the residue field O /λ, then there is a unique semisimple residual representation ρ : Gal(Q/Q) → GL2 (k) satisfying Tr(Frob(q)) = a q (g) ∈ k. We choose the primes λ (and `) such that the following conditions are met. To make sense of condition (5) below, we assume that the fields Q(χt ) and Q(µ p∞ ) are linearly disjoint. This is always the case if the class number of K is prime to p. We require that (1) the prime number ` be relatively prime to 2N Dp; (2) ` not divide the order of the Shimura subgroup, which is, by definition, the kernel of the natural map J0 (N ) → J1 (N ); (3) ` not divide the order of F∞ , where F∞ is as above; (4) the representation ρ be irreducible; and (5) ` split completely in Q(χt ), while remaining inert in Q(µ p∞ ). There are clearly infinitely many λ | ` satisfying these conditions. 6.4 Now we want to choose an auxiliary prime q for the purpose of raising the level. We select q such that (1) q - 2N Dp`, (2) q is inert in K , and (3) ρ(Frob(q)) = ρ(c), where c denotes a complex conjugation. Note that the third condition implies that X 2 − aq X + q ≡ X 2 − 1 (mod λ). In particular, we have q ≡ −1 (mod `). Under the above hypotheses, the following theorem of K. Ribet [Ri1] is fundamental.

252

V. VATSAL

THEOREM 6.5 P There exists a modular form h = bn (h)q n on 00 (N q) such that an ≡ bn (mod λ) for all (n, q) = 1 and such that bq = ±1.

Observe that h is of level M = N q, and so L(h, χ, s) has a functional equation with sign +1. Thus the results developed in the first part of this paper are applicable. As we have already remarked, our goal is to relate the nonvanishing of L(h, χ , 1) modulo λ to the indivisibility of classical Heegner points for g. The next section collects the necessary facts about Heegner points on modular curves. CM points on modular curves We note at the outset of this discussion that if A is an elliptic curve with complex multiplication by the quadratic field K , then we identify EndQ (A) ⊗ Q with K by fixing an embedding K ,→ Q, and an isomorphism Lie(A)Q ∼ = Q. Then the natural action of EndQ (A) ⊗ Q on LieQ (A) gives a map EndQ (A) ⊗ Q → Q which identifies EndQ (A) ⊗ Q with K . 6.6. Enhanced elliptic curves Recall that an enhanced elliptic curve over a field k is a pair (A, b), where A is an elliptic curve over k and where b ⊂ A is a k-rational subgroup of order N whose points over an algebraic closure of k form a cyclic group. (This terminology is due to Ribet [Ri2].) There is an evident notion of isomorphism classes of such enhanced elliptic curves. Each enhanced elliptic curve over k is by definition a k-rational point of the modular curve X 0 (N ). Now recall the usual Heegner points on X 0 (N ). Over C, these are pairs Q = (A, b), where A is an elliptic curve with complex multiplication by an order oc of conductor c in K , enhanced with a cyclic subgroup b of order N which is stable under the action of endomorphisms in oc . We assume throughout that N is prime to c. Let n ⊂ oc denote the annihilator of b in oc . Then b determines and is determined by n since b = A[n] ⊂ A is the set of elements killed by n. Note also that oc /n ∼ = Z/N Z. We sometimes write (A, n) instead of (A, b), and we say that the subgroup b is associated to the ideal n. In this situation, the theory of complex multiplication states that the point (A, b) on X 0 (N ) may be defined over the ring class field K (c). There are ec such Heegner points Q on X 0 (N ), where ec is the order of the group Pic(oc ), and the group Gal(K (c)/K ) ∼ = Pic(oc ) acts simply and transitively on the set of Heegner points. Let Q be a prime of Q whose residue characteristic q is prime to N Dc, where D is the discriminant of K , and such that q = Q ∩ K is inert in K . The curve A

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

253

admits a model over K (c) with good reduction at Q, and the reduction A of A at Q is a supersingular elliptic curve in characteristic q. According to classical results of Deuring, the supersingular elliptic curves in characteristic q correspond bijectively (via the endomorphism rings) to the set of conjugacy classes of oriented maximal orders in the definite quaternion algebra B ramified only at q and ∞. Thus A determines a class [A] (depending on Q) of maximal orders in B. On the other hand, the reduction b of b is a cyclic subgroup of order N in A. Thus the pair Q = (A, b) ∈ X 0 (N )(Fq 2 ) is an enhanced supersingular elliptic curve in characteristic q. The following proposition generalizes the classical results of Deuring. PROPOSITION 6.7 The enhanced supersingular elliptic curves in characteristic q are in one-to-one correspondence with the right ideal classes of any fixed Eichler order R of level N in B. Equivalently, the enhanced supersingular curves correspond to conjugacy classes of oriented Eichler orders of level N .

Proof This is [Ri2, Prop. 3.3]. The construction of the correspondence is recalled below. 6.8. Ribet’s construction In this section, we recall how reduction mod Q of CM points on X 0 (N ) produces Heegner points on quaternion algebras, in the sense previously considered in this paper. In other words, if we are given a pair (A, b), where A is an elliptic curve with complex multiplications by oc and where b is a cyclic subgroup of order N , we want to construct an oriented Eichler order R of level N , together with an oriented embedding f : Oc → R. Our discussion follows [Ri2, pp. 439 – 441]; for an alternative viewpoint, the reader may consult [C]. From now on, we assume that the conductor c is a power of p for a fixed prime p. This is adequate for our purposes in the rest of this paper. We also fix an ideal n of norm N in o K . Then n ∩ on is an ideal of index N in on . If A is a given elliptic curve with complex multiplications by on for n ≥ 0, we may view A as being enhanced with the cyclic subgroup b = A[n]. So let (A, b) be as above. The reduction (at Q) of endomorphisms gives an embedding f : on → R, where R ⊂ End(A) is the Eichler order of level N given as follows. If we let t : A → A/b denote the canonical quotient map, then there is a natural inclusion of End(A/b) into End(A)⊗Q ∼ = B given by φ 7→ t ◦φ◦t −1 . Then R is presented as the intersection

254

V. VATSAL

End(A) ∩ End(A/b). One checks readily that on → End(A) induces an embedding on → R. Note that this prescription allows us to associate to any enhanced supersingular curve (A, b) an Eichler order R of level N . This is the correspondence of Proposition 6.7, but we need to fix the orientations. In the present situation, this is easy. The fixed ideal n chosen at the outset gives a homomorphism o K → Z/N Z, which defines local orientations on o K at each (split) prime dividing N . This induces local orientations on each order on . Similarly, an arbitrary choice of local orientation at q on o K leads to an orientation at q on each on . We therefore orient the Eichler order R that we have constructed by requiring that the embedding on → R be an oriented embedding. Note however that the Eichler order R so constructed arises as a subring of the quaternion algebra End(E)⊗Q, and note that there is no canonical way to compare the Eichler orders associated to different elliptic curves. Thus we fix an enhanced elliptic curve (A, n) of conductor 1. For any elliptic curve A0 with CM by on , we choose a nonzero isogeny γ : A → A0 . Then t 7→ γ −1 ◦ t ◦ γ induces End(A)0 → End A and an isomorphism End(A)0 ⊗ Q → End(A) ⊗ Q. If R0 is the Eichler order associated to A0 (or A0 ), then this procedure identifies R0 with an oriented Eichler order of level N inside the quaternion algebra B = End(A) ⊗ Q. Owing to the indeterminacy in the choice of γ , the order R0 is only determined up to conjugacy in B × . In summary, given an enhanced elliptic curve (A, n), where A has complex multiplication by on and where b is associated to a fixed ideal n of norm N in o K , we have constructed a pair ( f, R), where R is an oriented Eichler order in a fixed realization B of the quaternion algebra of discriminant q and where f : on → R is an oriented embedding. The pair ( f, R) is a Heegner point on B as defined in the first part of this paper. For completeness, we review the construction of Ribet which leads to the proof of Proposition 6.7. As explained above, any enhanced elliptic curve (A, b) in characteristic q gives rise to an Eichler order R of level N inside the quaternion algebra B = End(A) ⊗ Q. We fix an enhanced elliptic curve (A, b) in characteristic q, together with the associated Eichler order R. Then the set of oriented Eichler orders of ˆ × Rˆ × . ˆ Q level N in B = End(A) ⊗ Q coincides with the double coset space B × \ B/ ˆ × Rˆ × = B × \ B/ ˆ Q ˆ Rˆ × is isomorphic to the set As we have already remarked, B × \ B/ of right ideal classes of R. Now let (A0 , b0 ) denote any other enhanced supersingular curve in characteristic q. Write Hom N (A, A0 ) for the subset of Hom(A, A0 ) consisting of homomorphisms that carry b to b0 . Then Hom N (A, A0 ) is a locally free right-R module, under composition with endomorphisms of A. Thus we can associate to the pair (Ao , b0 ) the class ˆ Rˆ × of the locally free right-R-module Hom N (A, A0 ). In terms of Eichler in B × \ B/ orders, we may associate to (A0 , b0 ) the left order of the right ideal Hom N (A, A0 ).

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

255

One checks without difficulty that if (A, n) is a CM point of conductor p n on X 0 (N ) (with a fixed choice of n) and if P = ( f, R) is the point on B constructed above, then the class of R in Cl(B) coincides with the class associated to the enhanced supersingular curve (A, b) by Ribet’s construction. 6.9. Action of the Picard group Let A denote an elliptic curve with complex multiplication by on , enhanced with a cyclic subgroup b, and let Q = (A, b) be the corresponding CM point on X 0 (N ). Then we have seen that reduction at Q gives rise to a pair P = ( f, R), where R is an oriented Eichler order of level N in the quaternion algebra B ramified at q and ∞. Thus P is a Heegner point on B, as in the first part of this paper. The next lemma verifies that the association Q 7→ P is compatible with the action of Pic(on ), but before stating the result, we also fix some notation. Given a point Q = (A, b), we have two distinct notations of reduction; namely, we have the geometric point x = x(Q) = (A, b) ∈ X 0 (N )(Fq 2 ) and the point P = ( f, R) on the definite quaternion algebra arising from the supersingular point x. It is convenient in the sequel to write Q to denote the point P since it is the point P that is of primary interest, and it is important to keep track of the dependence of P = Q on Q. 6.10 Let Q = (A, b) denote a CM point on X 0 (N ) as above. Let P = Q denote the corresponding point on the definite quaternion algebra B. Then, if σ ∈ Pic(On ), we have σ Qσ = Q , LEMMA

where σ acts on the left-hand side via the Artin map Pic(On ) → Gal(K ( p n )/K ), followed by the Galois action on geometric points, and the action on the right-hand side is the one from §2.2. Proof This is [BD3, Lem. 4.2]. Raising the level P Theorem 6.5 states that there exists a form h = bn (h)q n of level N q, new at q, such that an (g) ≡ bn (h) (mod λ) for (n, q) = 1, and bq (h) =  = ±1. The precise value of  is not relevant to our discussion. We let ψ denote the function considered in the first part of this paper, attached to the modular form g. We may now state the theorem that is our target in this section.

256

V. VATSAL

THEOREM 6.11 Let χ denote any anticyclotomic character of conductor p n with n ≥ 1. Let Q n denote a Heegner point on X 0 (N ) of conductor p n , and let Q˜ n denote its image under the quotient map J0 (N ) → E. Let Pn denote its reduction modulo Q as above. If the P P number σ χ (σ )ψ(Pnσ ) is a λ-adic unit, then the point σ χ (σ ) Q˜ σn is nonzero in E(Hn ) ⊗ k, and so in E(Hn ) ⊗ Q(χ).

As described in the introduction, the proof of Theorem 6.11 amounts to finding two different ways of describing the function ψ : Cl(B) → O /λ: one in terms of the modular curve X 0 (N ), and the other in terms of the Gross curve X of level M = N q. Furthermore, it is clear that Theorem 6.11, together with Theorem 1.2, implies Theorem 1.4. Construction of an unramified cover 6.12 We want to define the Galois cover of X 0 (N ) which is necessary for our construction. Let π : J0 (N ) → E denote the modular parametrization of E. We may assume that the kernel of π is connected, and stable under the Hecke operators, so that there is an induced action of the Hecke algebra on E. We let m denote the maximal ideal of T cut out by the form g modulo λ; it follows from the fact that ρ is irreducible that E[m] has dimension two over k = T/m. Furthermore, the action of Frob(q) on E[m]
0 . We let V = E[m], and we let V is given up to conjugation by the matrix 10 −1 ± denote the ±-eigenspace for Frob(q). Then V± is a finite flat group scheme over Fq , which becomes constant over Fq 2 . Now let E ± = E/V± . Then E ± is defined over Fq since the finite subgroup V± is Fq -stable. Let E ± → E denote the dual isogeny, so that the kernel W± of E ± → E is Cartier dual to V± . Note that, under our hypotheses, we have q ≡ −1 (mod `), so that µ` becomes constant over Fq 2 ; this implies that we have W± ∼ = V± over Fq 2 , and that, in fact, all these group schemes are constant. In particular, the covering map E ± → E is a Galois cover over Fq 2 with Galois group W± ∼ = k. A priori this is an isomorphism of additive groups, but note also that V± is stable under the action of the Hecke algebra, and note that, the module W± being dual to V± , it inherits the structure of a k-module as well. Now consider the composite X 0 (N ) → J0 (N ) → E. Pullback of the cover E ± → E gives a cover X ± → X 0 (N ), which has structural group W± , and so is Galois over Fq 2 . Let x ∈ X 0 (N )(Fq 2 ). Then the mechanism of the Frobenius substitution (see [S1, Chap. VI, §22]) gives an element F(x) ∈ Gal(X ± / X ) ∼ = W± . Thus we deduce a map F± : X 0 (N )(Fq 2 ) → W± .

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

257

This applies, in particular, when x is a supersingular point since, as is well known, all supersingular points are rational over Fq 2 . Note that the cover X ± , being the pullback of an isogeny, depends on the embedding X 0 (N ) → J0 (N ). We follow the standard procedure and take this embedding to be given by x 7→ (x − ∞). It should be pointed out that the cusp ∞ is rational over Fq , and that the fiber over ∞, being the group scheme W± , becomes constant over Fq 2 . It follows from this that the map F± is well defined on X 0 (N )(Fq 2 ). Fixing an isomorphism W± ∼ = k, writing 6 for the set of supersingular points on X 0 (N ), and extending F± by linearity, we obtain a morphism F± : Z[6] → k.

(15)

LEMMA 6.13 The following statements hold. (1) The map F± defined in (15) is nonzero and surjective. (2) If x ∈ X 0 (N )(Fq 2 ) is any point and if Tr is a Hecke operator with q 6= r , then we have F± (Tr x) = tr F± (x), where tr is the image of Tr in T/m. (3) If φ denotes the Frobenius element in Gal(Fq 2 /Fq ) and if x ∈ X 0 (N (Fq 2 )) is any point, then we have F± (φ(x)) = φ(F± (x)), where φ(q) acts on F± (x) ∈ W± via the usual Galois module structure.

Proof The first statement follows from a theorem of Ihara. Indeed, [I, Th. 1] shows that if X (N ) denotes the modular curve corresponding to the full congruence subgroup of level N , then the fundamental group of X (N )(Fq 2 ) is generated by the Frobenius elements of the supersingular points. Thus if X 0 → X (N ) is any connected unramified Galois cover over Fq 2 , then the reciprocity map Z[6(N )] → Gal(X 0 / X (N )) is nonzero and surjective, where 6(N ) denotes the set of supersingular points on X (N ). Note that the supersingular points on X (N ) are all rational over F(q 2 ) and that the natural projection X (N ) → X 0 (N ) takes 6(N ) to 6. 0 be the pullback to X (N ) of X → X (N ), where the cover X We let X 0 = X ± ± ± 0 of X 0 (N ) is as above, and the pullback is taken under the natural projection of X (N ) 0 is connected. Indeed, if this were not the case, then to X 0 (N ). Then we claim that X ± the kernel of J0 (N ) → J (N ) would have an element of order ` since X ± is a degree ` cover of X 0 (N ). But J0 (N ) → J (N ) factors as J0 (N ) → J1 (N ) → J (N ), and the latter map is injective since the cusp ∞ is totally ramified in X (N ) → X 1 (N ). Thus ker{J0 (N ) → J (N )} = ker{J0 (N ) → J1 (N )} is the Shimura subgroup, and we have assumed that ` is relatively prime to the order of this subgroup. 0 is connected, and Z[6(N )] → Gal(X 0 / X (N )) is nonzero and surjecThus X ± tive. But now it also follows from the fact that ` is prime to the order of the Shimura

258

V. VATSAL

subgroup that the cover X ± → X 0 (N ) is linearly disjoint from X (N ) → X 0 (N ). 0 / X (N )) is canonically isomorphic to Gal(X / X (N )). Since 6(N ) Thus Gal(X ± ± 0 projects to 6 and since all these points are rational over Fq 2 , the first statement now follows from the functorial properties of the reciprocity map. As for the second, select y ∈ J0 (N )(Fq ) such that `y = x. Then, if π± : E → E ± = E/V± is the quotient map and if π±∗ is the dual, we have π±∗ ◦π± (y) = `y = x, so that if z = π± (y), then π±∗ (z) = x. If φ 0 = φ q denotes the Frobenius element in Gal(Fq /Fq 2 ), then z 0 = φ 0 (z) also satisfies π±∗ (z 0 ) = x since x is fixed by φ 0 . Thus z 0 − z ∈ ker(π±∗ ), and one checks without difficulty that F± (x) = z 0 − z ∈ W± (see also [S1, Chap. 6, §23]). On the other hand, we also have `Tr y = Tr x. Since Tr is defined over Fq , we find that
φ 0 (Tr y) − Tr y = Tr φ 0 (y) − y ∈ J0 (N )[`]. The required statement follows upon applying π± . Care has to be taken with one point, namely, that the action of T on W± , viewed as a quotient of J0 (N )[`], coincide with the one induced by duality from V± . But this follows from autoduality of J0 (N )[`] under the twisted Weil pairing. (Note that the Atkin-Lehner involution acts as ±1 on J0 (N )[m].) The final statement may be proven in a similar fashion. Choosing z ∈ E ± such that π±∗ (z) = x, we have π± (φ(z)) = φ(x), so that


φ F± (x) = φ φ 0 (z) − z = φ 0 φ(z) − φ(z). 6.14 Now let Q = (A, b) ∈ X 0 (N )(Hn ) denote a Heegner point, as above. Recall our convention that x denotes the geometric point (A, b) ∈ X 0 (N )(Fq 2 ) and that Q denotes the point deduced from x on the quaternion algebra B. Applying F± to the point x, we obtain F± (x) = F± (Q) ∈ W± . It is clear that if F± (Q) 6= 0, then the image of x in E(Fq 2 ) is nonzero in E(Fq 2 ) ⊗ k. In view of our choice of `, we find that Q˜ has infinite order in E(Hn ). Indeed, the following slightly stronger lemma is obvious. LEMMA 6.15 Let Q = (A, b) be as above, and let χ denote any character of Gal(Hn /K ). Then, P P if σ χ (σ )F± (Q σ ) is nonzero in W± ⊗ k(χ), the point σ χ (σ ) Q˜ σ is nonzero in E(Hn ) ⊗ Q(χ ).

6.16. Proof of Theorem 6.11 To complete the proof, we relate the functions F± to certain functions ψ coming from definite quaternion algebras. Let h be a form of level M = N q, congruent to g, as

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

259

above. Let X denote the Gross curve of level N , associated to the quaternion algebra B ramified at ∞ and q. Let M denote the group Pic(X ). It follows from Proposition 6.7 that M may be identified with the free Z-module on the set 6 of supersingular points on the modular curve X 0 (N ). Furthermore, if T(M) denotes the Hecke algebra of level M = N q acting on M = Z[6], then one can relate the action of the Hecke operators in T(M) to those in T = T(N ) as follows. For clarity, we write TrM or TrN to denote the r th Hecke operator at levels M and N , respectively. With this notation, it is known that if r is prime to q, then the action of TrM on M ∼ = Z[6] is induced from the action of TrN acting on 6(N ). For a discussion of this, we refer the reader to [Ri2, pp. 444 – 445]. On the other hand, one knows (see [Ri2, Prop. 3.8]) that Tq (M) acts on Z[6(N )]0 via the Frobenius automorphism x 7→ φ(x) of 6(N ), where the superscript zero denotes the subgroup of divisors of degree zero. Let ψ = ψh : M → O denote the homomorphism associated to h, as in the introduction. If x ∈ Z[6]0 , we may view x as an element of M . Then we have ψ(Tr (x)) = br (h)ψ(x) for the eigenvalue br (h) of Tr = Tr (M) acting on h. Furthermore, we have ψ(Tq (x)) = ψ(x), where bq (x) =  = ±1. It follows from a multiplicity-one theorem of Mazur (see [Ri2, Th. 6.4]) that, up to unit multiples, there is a unique such nonzero homomorphism M 0 → F` . Choosing the sign α = ± judiciously and scaling by a unit if needed, we find, from Lemma 6.13, that the function Fα satisfies the same properties of Hecke invariance, so that we get Fα = unit · ψ (mod λ). Theorem 6.11 is now an immediate consequence. Acknowledgments. It is a pleasure to thank H. Darmon for some useful conversations on the subject of this paper. I am also indebted to C. Cornut for his careful examination of an earlier version of this paper. The formulation of Proposition 5.3 borrows freely from Cornut’s work in [C]. I also thank an anonymous referee for some useful suggestions. References [BD1] [BD2] [BD3] [BD4]

[BD5]

M. BERTOLINI and H. DARMON, Kolyvagin’s descent and Mordell-Weil groups over

ring class fields, J. Reine Angew. Math. 412 (1990), 63 – 74. MR 91j:11048 224 , Heegner points on Mumford-Tate curves, Invent. Math. 126 (1996), 413 – 456. MR 97k:11100 221, 222, 224, 227, 231, 250 , A rigid analytic Gross-Zagier formula and arithmetic applications, Ann. of Math. (2) 146 (1997), 111 – 147. MR 99f:11079 255 , Heegner points, p-adic L-functions, and the Cerednik-Drinfeld uniformization, Invent. Math. 131 (1998), 453 – 491. MR 99f:11080 227, 232, 233, 244 , Euler systems and Jochnowitz congruences, Amer. J. Math. 121 (1999), 259 – 281. MR 2001d:11060 224, 225

260

[BD6] [C] [DT] [FW] [G]

[Gr]

[H1] [H2] [I]

[J]

[M]

[P]

[R] [Ri1]

[Ri2] [S1]

[S2] [St]

V. VATSAL

, p-adic periods, p-adic L-functions, and the p-adic uniformization of Shimura curves, Duke Math. J. 98 (1999), 305 – 334. MR 2000f:11075 227, 232, 233 C. CORNUT, Mazur’s conjecture on higher Heegner points, Invent. Math. 148 (2002), 495 – 523. CMP 1 908 058 225, 239, 240, 241, 253, 259 F. DIAMOND and R. TAYLOR, Nonoptimal levels of mod ` modular representations, Invent. Math. 115 (1994), 435 – 462. MR 95c:11060 240, 243 B. FERRERO and L. C. WASHINGTON, The Iwasawa invariant µ p vanishes for abelian number fields, Ann. of Math. (2) 109 (1979), 377 – 395. MR 81a:12005 222, 244 R. GREENBERG, “Iwasawa theory for p-adic representations” in Algebraic Number Theory, Adv. Stud. Pure Math. 17, Academic Press, Boston, 1989, 97 – 137. MR 92c:11116 223 B. H. GROSS, “Heights and the special values of L-series” in Number Theory (Montreal, 1985), CMS Conf. Proc. 7, Amer. Math. Soc., Providence, 1987, 115 – 187. MR 89c:11082 229 H. HIDA, Modules of congruence of Hecke algebras and L-functions associated with cusp forms, Amer. J. Math. 110 (1988), 323 – 382. MR 89i:11058 229, 230 , The Iwasawa µ-invariant of p-adic Hecke L-functions, preprint, 2002, http://math.ucla.edu/˜hida 222 Y. IHARA, “On modular curves over finite fields” in Discrete Subgroups of Lie Groups and Applications to Moduli (Bombay, 1973), Oxford Univ. Press, Bombay, 1975, 161 – 202. MR 53:2956 257 N. JOCHNOWITZ, “A p-adic conjecture about derivatives of L-series attached to modular forms” in p-adic Monodromy and the Birch and Swinnerton-Dyer Conjecture (Boston, 1991), Contemp. Math. 165, Amer. Math. Soc, Providence, 1994, 239 – 263. MR 95g:11037 224 B. MAZUR, “Modular curves and arithmetic” in Proceedings of the International Congress of Mathematicians (Warsaw, 1983), Vol. 1, 2, PWN, Warsaw, 1984, 185 – 211. MR 87a:11054 224 B. PERRIN-RIOU, Fonctions L p-adiques associ´ees a` une forme modulaire et a` un corps quadratique imaginaire, J. London Math. Soc. (2) 38 (1988), 1 – 32. MR 89m:11043 221 M. RATNER, Raghunathan’s conjectures for Cartesian products of real and p-adic Lie groups, Duke Math. J. 77 (1995), 275 – 382. MR 96d:22015 248, 249 K. A. RIBET, “Congruence relations between modular forms” in Proceedings of the International Congress of Mathematicians (Warsaw, 1983), Vol. 1, 2, PWN, Warsaw, 1984, 503 – 514. MR 87c:11045 251 , On modular representations of Gal(Q/Q) arising from modular forms, Invent. Math. 100 (1990), 431 – 476. MR 91g:11066 252, 253, 259 J.-P. SERRE, Groupes alg´ebriques et corps de classes: Cours au Coll`ege de France, Publ. Inst. Math. Univ. Nancago 7, Actualit´es Sci. Indust. 1264, Hermann, Paris, 1959. MR 21:1973 256, 258 , Trees, Springer, Berlin, 1980. MR 82c:20083 242, 249 G. STEVENS, Stickelberger elements and modular parametrizations of elliptic curves, Invent. Math. 98 (1989), 75 – 106. MR 90m:11089 223, 230

SPECIAL VALUES OF ANTICYCLOTOMIC L -FUNCTIONS

[V1] [V2]

[W] [Z]

261

V. VATSAL, Canonical periods and congruence formulae, Duke Math. J. 98 (1999),

397 – 419. MR 2000g:11032 224, 229, 230 , Uniform distribution of Heegner points, Invent. Math. 148 (2002), 1 – 46. CMP 1 892 842 219, 220, 221, 222, 224, 226, 227, 232, 233, 236, 238, 239, 245, 248, 249 L. C. WASHINGTON, The non- p-part of the class number in a cyclotomic Z p -extension, Invent. Math. 49 (1978), 87 – 97. MR 80c:12005 222 S.-W. ZHANG, Gross-Zagier formula for GL2 , Asian J. Math. 5 (2001), 183 – 290. CMP 1 868 935 224

Department of Mathematics, University of British Columbia, Vancouver, British Columbia V6T 1Z2, Canada; [email protected]

SINGULAR LAGRANGIAN MANIFOLDS AND SEMICLASSICAL ANALYSIS ` YVES COLIN DE VERDIERE

Abstract Lagrangian submanifolds of symplectic manifolds are very central objects in classical mechanics and microlocal analysis. These manifolds are frequently singular (integrable systems, bifurcations, reduction). There have been many works on singular Lagrangian manifolds initiated by V. Arnold, A. Givental, and others. The goal of our paper is to extend the classical and semiclassical normal forms of completely integrable systems near nondegenerate (Morse-Bott) singularities to more singular systems. It turns out that there is a nicely working way to do that, leading to normal forms and universal unfoldings. We obtain in this way natural ansatzes extending the Wentzel-Kramers-Brillouin(WKB)-Maslov ansatz. We give more details on the simplest non-Morse example, the cusp, which corresponds to a saddle-node bifurcation. Contents 0. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1. Singular Lagrangian manifolds . . . . . . . . . . . . . . . . . . . . . . 1.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3. Phase functions . . . . . . . . . . . . . . . . . . . . . . . . . . 2. Infinitesimal deformations . . . . . . . . . . . . . . . . . . . . . . . . 3. Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4. Integrable systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1. Singularities of integrable systems . . . . . . . . . . . . . . . . . 4.2. Singularities of integrable systems and deformations of Lagrangian manifolds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3. Generic singularities of integrable systems with 2 degrees of freedom 5. The symplectic codimension of curves with isolated singularities . . . . . 5.1. Vanishing cohomology: A short review . . . . . . . . . . . . . . . 5.2. Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 2, Received 4 May 2001. Revision received 7 December 2001. 2000 Mathematics Subject Classification. Primary 32S05, 32S40, 34E20, 35P20, 53D12, 70H06.

263

264 266 266 266 268 269 271 272 272 272 273 274 274 275

264

6. 7. 8.

` YVES COLIN DE VERDI ERE

Normal forms . . . . . . . . . . . . . . . . . . . . . . . . . . Versal deformations: The formal case . . . . . . . . . . . . . . . The quasi-homogeneous case . . . . . . . . . . . . . . . . . . . 8.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 8.2. Using Euler identity . . . . . . . . . . . . . . . . . . . . 9. Versal deformations for quasi-homogeneous singularities . . . . . 9.1. A remarkable identity . . . . . . . . . . . . . . . . . . . 9.2. Nonvanishing of the Jacobian determinant of action integrals 9.3. Lifting isotopies . . . . . . . . . . . . . . . . . . . . . . 9.4. Versal deformation theorem in the holomorphic case . . . . 10. Semiclassics . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1. Semiclassical normal forms . . . . . . . . . . . . . . . . 10.2. Mixed case . . . . . . . . . . . . . . . . . . . . . . . . 10.3. The case of quasi-homogeneous singularities . . . . . . . . 11. Singular Bohr-Sommerfeld rules: The general scheme . . . . . . 11.1. The context . . . . . . . . . . . . . . . . . . . . . . . . 11.2. Local models and scattering matrices . . . . . . . . . . . . 11.3. Singular holonomies . . . . . . . . . . . . . . . . . . . . 11.4. Regularization . . . . . . . . . . . . . . . . . . . . . . . 11.5. Singular Bohr-Sommerfeld rules . . . . . . . . . . . . . . 12. The cusp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1. Classics . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2. Semiclassics . . . . . . . . . . . . . . . . . . . . . . . . 12.3. Computation of the first coefficients a1,0 and b1,0 . . . . . . 12.4. The model problem . . . . . . . . . . . . . . . . . . . . 12.5. The semiclassical bifurcation . . . . . . . . . . . . . . . . 12.6. Bohr-Sommerfeld rules . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

277 277 279 279 279 280 280 281 282 284 285 285 286 286 287 287 287 289 289 290 290 291 291 291 292 294 295 295

0. Introduction Papers [11], [12], [13], [14], [15], and [37] considered semiclassical completely integrable Hamiltonian systems whose singularities are of Morse-Bott type using normal forms of Birkhoff type. In the nice paper [34], which was an important source of inspiration for us, F. Pham showed the universality of solutions of semiclassical Schr¨odinger equations with polynomial potentials. Our goal is to extend this analysis allowing (more general) canonical transformations in order to study, for example, • the saddle-node bifurcation, • the Birkhoff normal form in the case of k : 1 resonances with k ≥ 3 in the spirit of [15],

SINGULAR LAGRANGIAN MANIFOLDS

265

the bifurcation of periodic orbits of a Hamiltonian system where the Poincar´e map of a periodic orbit admits an eigenvalue that is a cubic root of 1, • the adiabatic limit or Born-Oppenheimer approximation with crossings of more than 2 eigenvalues. This way, we propose a general setting inspired by Thom’s catastrophe theory (see [4]) and present a sketchy study of the saddle-node bifurcation (cusp) ξ 2 + x 3 = 0. A more algebraic (cohomological approach) is presented in [41]. The subject is really the study of the singularities of Lagrangian manifolds, of their deformations (or bifurcations), and of the associated semiclassical ansatzes. Building up classical and semiclassical normal forms leads to studying model problems depending on a finite number of parameters, among which the simplest have already been described in the literature: cubic oscillators (see [10], [9], [20]), quartic oscillators (see [19]), and polynomial potentials (see [43]). A noticeable fact is that we can use the same methods for classical and semiclassical bifurcations, and in particular, the codimensions of the singularities are the same. Of course, the study of the classical Hamiltonian dynamic in a 2-dimensional phase space is trivial, but this is no longer true for the semiclassical case, which we reduce to special functions. The reader should take note of the fact that caustic singularities are a different problem for which Lagrangian manifolds are usually smooth. We use strongly canonical transformations that eliminate the problem of caustics. The main idea is to forget the equations of the manifolds and to focus on the ideal of functions which vanish on it. The same idea turned out to be very important in algebraic geometry. On the quantum side, we do the same change of point of view: we consider left ideals of pseudodifferential operators. We can do that because any b = 0 also satisfies b b = 0 for any operator b solution of Pu B Pu B. It appears that the usual singularities, at least for 1 degree of freedom, do admit normal forms and that their deformations have a universal model depending on a finite number of parameters. The solutions of this model are the ad hoc special functions: the smooth case corresponds in that way to the WKB-Maslov ansatz; the Morse-Bott case corresponds to Lagrangian intersections (hyperbolic case) or to coherent states (elliptic case). An important part of the program is the study of these special functions. In the case of the cusp ξ 2 + x 3 = 0, it is enough to study the differential equation (cubic Schr¨odinger equation) •

−u 00 + (x 3 + Ax + B)u = 0. We give the general definitions for any dimension, and after Section 2 we restrict to the case of a 2-dimensional phase space. The main nontrivial result is Theorem 6, which is a holomorphic versal deformation result for quasi-homogeneous isolated singularities of curves.

266

` YVES COLIN DE VERDI ERE

The semiclassical results then follow from the techniques already developed in [14].

1. Singular Lagrangian manifolds 1.1. Definitions There are several possible definitions of germs of singular Lagrangian manifolds. We work in the real analytic context. We denote by (Z 2d , ω; z 0 ) a germ of a nonsingular real analytic symplectic manifold of dimension 2d which, by the Darboux theorem, P can be identified with (T ? R d , dξi ∧ d xi ; 0). E denotes the algebra of germs of real-valued analytic functions (or smooth functions). Definition 1 (1) A (germ of a) singular Lagrangian manifold L in Z 2d is a germ of a real analytic variety (i.e., complex variety invariant by complex conjugation) of dimension d which is Lagrangian near all smooth points. We denote by L or L L the ideal of E of functions vanishing on L. If F j , j = 1, . . . , n is a system of generators of L , we denote L = hF1 , . . . , Fn i. This ideal is involutive, meaning that {L , L } ⊂ L . (2) A (germ of a) singular Lagrangian manifold L is a complete intersection if the ideal L L is generated by d functions. (3) A (germ of a) singular Lagrangian manifold L is a singular leaf of a Lagrangian foliation if L L admits a set of generators F j , j = 1, . . . , d, such that {F j , Fk } = 0 for all j, k. In the first case we speak about a (germ of a) singular Lagrangian manifold, in the second of a (germ of a) singular Lagrangian manifold that is a complete intersection, and in the third of a (germ of a) singular leaf of a completely integrable system. We can ask the following. Question 1 Are cases (2) and (3) really distinct? Does every singular Lagrangian manifold that is a complete intersection admit Poisson commuting generators F j , j = 1, . . . , d ? 1.2. Examples Example 1 Let d = 1, and let f : Z 2 → R be an analytic map. If zero is a critical value of f , the curve { f = 0} is a Lagrangian singular manifold with respect to all possible

SINGULAR LAGRANGIAN MANIFOLDS

267

definitions. If f is a Morse function, the level sets L E = f −1 (E) are smooth except for a discrete set of energies. Example 2 Let us start with an anharmonic oscillator with only one resonance, like H = |z 1 |2 + P |z 2 |2 + dj=3 ω j |z j |2 + O(|z|3 ), where z j = x j + iξ j and (1, ω3 , . . . , ωd ) are linearly independent over the rationals. We get an integrable system using the truncated Birkhoff normal form. The Hamiltonians are F1 = |z 1 |2 + |z 2 |2 , F2 = |z 3 |2 , Fd−1 = |z d |2 , Fd = K , where K = O(|z|3 ) is a polynomial. Reducing by the action of T d−1 given by the d − 1 first Hamiltonians, we get a complex projective line depending on d − 1 parameters; K can be seen as a function on this projective line depending on d − 1 parameters, and hence the Lagrangian foliation admits generically all singularities of codimension less than or equal to d of functions of 2 variables. (This example was described to me by M. Joyeux; see Section 4.2.) Example 3 The “normal bundle” L of the cusp 9x 2 − 8y 3 = 0, namely, the closure of the normal bundle to the nonsingular part of it, is a singular Lagrangian manifold parametrized by  u3 u2  m(u, v) = , ; v, −uv . 3 2 The ideal L of functions vanishing on L is minimally generated by F1 = 9x 2 − 8y 3 ,

F2 = 3xξ + 2yη,

F3 = η2 − 2yξ 2

F4 = 3xη + 4y 2 ξ

(as computed by M. Morales); hence it is not a complete intersection. Example 4 The (open) swallowtail S (see [1]) can be defined as the subset S of the set Z of polynomials P = x 5 + ax 3 + bx 2 + cx + d admitting a zero of order at least 3. We can write P = (x − u)3 (x 2 + 3ux + v), which gives a parametrization of S. There exists a natural symplectic structure on Z for which S is Lagrangian. This manifold is obtained in a generic way in the following problem: If X ⊂ R3 is a surface and V is a vector field on X whose integral curves are geodesics, the set of affine lines generated by the vectors V (m), m ∈ X , is a (singular) Lagrangian manifold in the symplectic manifold of affine lines in R3 . It can be shown that S is not a complete intersection.∗ ∗I

thank Marcelo Morales very much for the computations of these examples.

` YVES COLIN DE VERDI ERE

268

1.3. Phase functions The WKB-Maslov ansatz allows us to associate to any smooth Lagrangian submanifold of T ? Rn a family of h-dependent functions u h (x) (h is a small positive parameter) given by oscillatory integrals of the form Z u h (x) = eiϕ(x,θ )/ h a(x, θ) dθ RN

whose microsupport is the reduced Lagrangian manifold  L ϕ = (x, ∂x ϕ) ∂θ ϕ = 0 . Locally, every Lagrangian submanifold of T ? Rn can be defined by the previous formula using a so called nondegenerate phase function ϕ (see [22, page 31]). This construction is a special case of symplectic reduction. In order to do the same thing for a singular Lagrangian manifold, one could try to use degenerate phase functions. Definition 2 Let L ⊂ T ? Rn be a germ of a singular Lagrangian manifold whose singular part is denoted by L 0 and whose smooth n-dimensional stratum is denoted by L 1 . A germ of a smooth function ϕ : Rn × R N → R is a phase function for L if the map jϕ : Cϕ → L, where Cϕ = {(x, θ)|∂θ ϕ = 0} and jϕ (x, θ) = (x, ∂x ϕ), is a homeomorphism and ϕ restricted to the open set of Cϕ , where the ∂θ j ϕ, j = 1, . . . , N , are independent, is a nondegenerate phase function for L 1 . LEMMA 1 If L admits a phase function, the Maslov index of any loop included in L 1 vanishes.

Proof The proof is a consequence of the fact that the Maslov index can be defined (see ˇ [26, pages 154 – 163] and also the appendix by Arnold in the book [30]) in Cech cohomology by the cocycle ind(∂θ,θ ϕi ) − ind(∂θ,θ ϕ j ) (ind(q) is the Morse index of the quadratic form q), which in our case gives a trivial cocycle because there is only one open set in the covering. Question 2 What is a characteristic property of singular germs of Lagrangian manifolds which admit a degenerate phase function? With respect to this question, we propose the following example.

SINGULAR LAGRANGIAN MANIFOLDS

269

Example 5 Let ϕ(x, θ ) = θ(x 2 − θ 2 /3). We get L ϕ = {ξ 2 − 4x 4 = 0}, so ϕ is a degenerate phase function for L ϕ . Example 6 We have the following (see also [39]). PROPOSITION 1 The germ at zero of the normal bundle of the cusp (Example 3) does not admit a degenerate phase function.

Proof The Maslov index of any closed curve inside the smooth part of the germ would be zero by Lemma 1. Let us consider the curve γ (θ) = m(cos θ, sin θ). The Lagrangian vector space tangent to L at the point γ (θ) is generated by the vectors (0, 0; 1, − cos θ), (cos2 θ, cos θ; 0, − sin θ), and by reduction with respect to ξ = 0, we get the curve θ → [(cos θ ; − sin θ)] inside the Lagrangian Grassmanian of T ? R whose Maslov index is ±2. 2. Infinitesimal deformations We propose below a very na¨ıve approach, restricting ourselves to phase spaces of dimension 2; a more precise and algebraic approach in any dimension can be found in [41]. We restrict ourselves in what follows to the case d = 1 (except in Sections 3 and 4). In this case, every curve is Lagrangian and is a complete intersection. Moreover, canonical transformations are just orientation- and area-preserving diffeomorphisms. Definition 3 We say that the germs of Lagrangian manifolds (hF0 i, ω0 ) and (hFi, ω) are equivalent if there exists a germ of diffeomorphism χ such that F ◦ χ = E F0 (E(0) 6= 0) and χ ? (ω) = ω0 . By the Darboux theorem, we often restrict ourselves to ω = ω0 . One can take a stronger form of equivalence by asking that F ◦ χ = ψ ◦ F0 (as in the paper Le lemme de Morse isochore [16]), where ψ is a germ of diffeomorphism of R fixing zero. One is then led to the same space of infinitesimal deformations, but it is inappropriate for our semiclassical business because it forces us to use functional calculus for operators that are in general non-self-adjoint.

` YVES COLIN DE VERDI ERE

270

We want to define the codimension of the set of equivalent germs of Lagrangian manifolds inside the set of all germs, so we need first to define infinitesimal deformations of germs. Definition 4 Given a singular germ of curve L in T ? R given by F0 = 0, the space of infinitesimal deformations (as a Lagrangian manifold) of L = hF0 i is the space of all germs of functions E . A general deformation of (F0 , ω0 ) is given by (Ft , ωt ). Using the Darboux theorem, we can reduce to deformations (F0 + t K + O(t 2 ), ω0 ). K is an arbitrary germ of a real-valued function. Definition 5 A deformation Lt = hFt i is trivial if there exists a smooth family χt of canonical transformations and a smooth family of functions E t ∈ E , such that Ft ◦ χt = E t F0 . This implies that there exist germs of functions X and Y such that the infinitesimal deformation K = ddtFt |t=0 satisfies K = {X, F0 } + Y F0 . We can now give the definition of the codimension of a germ of a Lagrangian curve. Definition 6 The codimension µ = µ(hF0 i, ω0 ) of the Lagrangian curve L = hF0 i is defined by
µ = dim E / ({E , F0 } + E · F0 ) , (1) where {·|·} is the Poisson bracket. If µ is finite, any basis K α ∈ DL , α = 1, . . . , µ, of a supplementary space of {E , F0 } + E · F0 in DL defines the (uni)versal deformation of L as follows: µ D E X F0 + aα K α . α=1

More precisely, we ask that equation (1) be true with E (U j ) for a basis U j of neighbourhoods of O (with the same functions K α ). Question 3 What is a natural extension of Definition 1 to the case of systems of operators, that is, matrix-valued germs of functions (see [8])?

SINGULAR LAGRANGIAN MANIFOLDS

271

3. Cases (1) The smooth case: The differentials d F j are linearly independent in some neighbourhood of the origin. Then L is a germ of a smooth Lagrangian manifold. This L is of codimension zero. Moreover, the Darboux theorem implies that, up to canonical transformation, L = hξ1 , . . . , ξd i. (2) The Morse (d = 1) case: Let Fε = F0 + O(ε), where F0 is a nondegenerate quadratic form on T ? R. By Le lemme de Morse isochore [16], there exist χε a germ of canonical transformations smoothly depending on ε and a smooth function 8ε such that Fε ◦ χε = 8ε ◦ F0 and 800 (0) 6= 0. Hence 8ε admits a nondegenerate zero t (ε) and we have
Fε ◦ χε (x, ξ ) = E ε (x, ξ ) F0 (x, ξ ) − t (ε) , from which it is clear that hF0 − ti is a versal deformation of hF0 i. (3) The Eliasson case (see [23] or the nondegenerate case of [38, d´efinition 2.1]): It is an extension of the previous case to several quadratic forms. Let q1 , . . . , qd be d independent commuting quadratic forms on T ? Rd , where (q1 , . . . , qd ) is of type (m e , m h , m f ), and d = m e + m h + 2m f , where m e is the number of elliptic forms, m h is the number of hyperbolic ones, and m f is the number of focus-focus ones. We have µ = d. This value is minimal for a rank-zero singular point of an integrable system. (4) Cusp (A2 ): We have F0 = ξ 2 + x 3 (d = 1) and µ = 2: K 1 = 1,

K 2 = x.

We see that up to canonical transformation any F that admits a nondegenerate cusp is equivalent to the standard example ξ 2 + x 3 . 2 4 (5) Quartic oscillator (A+ 3 ): We have F0 = ξ + x (d = 1) and µ = 3: K 1 = 2 1, K 2 = x, K 3 = x . 2 4 2 (6) Quartic antioscillator (A− 3 ): We have F0 = ξ − x or F0 = x(x − ξ ) (d = 1) and µ = 3. (7) Triple crossing (D4− ): We have F0 = xξ(x − ξ ) (d = 1) and µ = 4: K 1 = 1,

K 2 = x,

K 3 = ξ,

K 4 = xξ.

(8) Hyperbolic umbilic (D4+ ): We have F0 = x(x 2 + ξ 2 ) (d = 1) and µ = 4. Problem 1 Describe all singular Lagrangian manifolds of small codimension.

` YVES COLIN DE VERDI ERE

272

4. Integrable systems 4.1. Singularities of integrable systems Definition 7 Let (Z ; ω) be a germ of a symplectic manifold of dimension 2d; a germ of a completely integrable system is given by d germs of real functions F j , j = 1, . . . , d, such that the Poisson brackets {Fi , F j } all vanish and the differentials of the F j ’s are linearly independent almost everywhere. The map z → (F j (z)) of Z into Rn is called the momentum map. A singular point z 0 is a point where the rank r (z 0 ) of the d F j (z 0 )’s (1 ≤ j ≤ n) is less than d. The separatrix is the image by the momentum map (F j ) of the set of points z, where r (z) < n. 4.2. Singularities of integrable systems and deformations of Lagrangian manifolds Let hF(x, ξ ; t) = 0i ((x, ξ ) ∈ T ? R, t ∈ R N ) be a deformation of the germ of a (Lagrangian) curve hF(x, ξ, 0) = 0i, and assume ∂F (0, 0; 0) 6= 0. ∂t

(?)

We can associate to it a germ of a completely integrable system in T ? (R(tN0 ,x) ) in the following way. We choose coordinates t = (t 0 , t N ) such that ∂t N F 6= 0. Then we can rewrite F(x, ξ, t) = E(x, ξ, t)(H N (x, ξ ; t 0 ) − t N ). We take the commuting Hamiltonians t1 , . . . , t N −1 , H N on T ? (R(tN0 ,x) ) which define an integrable germ. We can go back to the deformation in the following way. We start with the integrable germ with a singularity of rank N −1 and choose t1 , . . . , t N −1 commuting integrals whose differentials at the singular point are independent. We reduce the systems, and we get for each a ∈ R N −1 , b ∈ R, a 2-dimensional curve hH N (x, ξ, a) = bi which gives the previous deformation. 2 The previous correspondence is an isomorphism between germs of integrable systems of rank N − 1 (modulo canonical diffeomorphisms) and N -parameter deformations of curves (modulo canonical diffeomorphisms) satisfying (?). PROPOSITION

Moreover, we get in that way a correspondence between universal deformations (deformations containing the versal deformation) and stable germs of integrable systems. A germ of integrable systems is said to be stable if any small perturbation of the germ of an integrable system is equivalent to the unperturbed system by a germ of a canonical transformation.

SINGULAR LAGRANGIAN MANIFOLDS

273

E

EE

L

E

2 tori 1 torus

C

EE

E

C H

E H EE

EH

E

Figure 1. Typical bifurcation diagram for a 2 degrees of freedom system

We see, using the previous correspondence, that universal deformations of a germ of a curve of codimension µ ≤ N correspond to stable singularities of integrable systems with N degrees of freedom. The image of the set of equivalent singularities by the momentum map is then of codimension N − µ. 4.3. Generic singularities of integrable systems with 2 degrees of freedom From the previous sections, we get the following list of locally stable singularities of integrable systems with 2 degrees of freedom (see [24] and Figure 1 for pictures of these separatrices for classical systems). (1) Rank 1: We have the following: E (elliptic) (x12 + ξ12 , ξ2 ) (µ = 1), H (hyperbolic) (x1 ξ1 , ξ2 ) (µ = 1), C (cusp) (x13 + ξ12 + x1 ξ2 , ξ2 ) (µ = 2). This list is the list of codimension less than or equal to 2 germs of plane curves which are obtained by reduction using the procedure described in Section 4.2. (2) Rank 0: We have the following: EE (elliptic-elliptic), EH (elliptic-hyperbolic), HH (hyperbolic-hyperbolic), L (loxodromic) (µ = 2). From the semiglobal picture (see [33]), we get other stable singularities that correspond to codimension 2 singularities in the Zm -equivariant cases:

` YVES COLIN DE VERDI ERE

274

(1) (2) (3) (4)

m m m m

= 2: x 4 ± ξ 2 ; = 3: <(z 3 ); = 4: (x 2 + ξ 2 )2 + a<(z 4 ) with a ∈ R \ {0, ±1}; > 4: (x 2 + ξ 2 )2 + a<(z m ) with a ∈ R.

Question 4 Are there other stable singularities? Find the corresponding list for d = 3, 4, . . . .

5. The symplectic codimension of curves with isolated singularities 5.1. Vanishing cohomology: A short review All results below are described in [29]. Let F0 : (C2 , 0) → (C, 0) be a germ of a holomorphic function, and assume that the origin is an isolated critical point of multiplicity µ0 or, equivalently, that if M is the (maximal) ideal of germs vanishing at zero, there exists k such that M k ⊂ J (F0 ), where J (F0 ), the jacobian ideal of F0 , is the ideal generated by the partial derivatives of F0 and dimC E /J (F0 ) = µ0 . Then, if ε > 0 is small enough and r = r (ε) > 0 is small enough, the map (x, y) → t = F0 (x, y) is a smooth fibration of B(0, ε) ∩ {0 < |F0 (x, y)| < r } on D ? = {0 < t < r } whose fiber X t is a Riemann surface. The vanishing homology is the family of homologies of X t which is a vector bundle on D ? . It has been proved by J. Milnor (see [31]) that X t has the homotopy type of a bouquet of µ0 circles, so that the vanishing homology H1van (X t ) is a vector space of dimension µ0 . It is generated by µ0 cycles γ j (t), which can be chosen locally constant with regard to t. Globally, when t goes around the origin, we get a monodromy that preserves the lattice generated by the geometric cycles. In order to make computations, it is useful to introduce the vanishing cohomology 1 Hvan as follows: we put 1 Hvan = 1 /(0 d F0 + d0 ),

where  j is the space of germs of holomorphic differential forms of degree j. One 1 is a free module of rank µ0 over A, where A = C{F } is the ring can prove that Hvan 0 of convergent series in F0 . We have the following. PROPOSITION 3 1 as a module over A, the restriction of the ω ’s to X for If (ω j ) is a basis of Hvan j R t t ∈ D ? is a basis of the cohomology of X t and the determinant of the matrix ( γi (t) ω j ) does not vanish.

SINGULAR LAGRANGIAN MANIFOLDS

275

The problem is now to find such a basis (ω j ). For that purpose, we introduce the map ω → ω ∧ d F0 ; it can be shown that this map induces an isomorphism of A-modules 1 to d F ∧ 1 /d F ∧ d0 . The space d F ∧ 1 is of finite codimension from Hvan 0 0 0 µ0 over C inside 2 , and hence, if we take the tensor product with the field M of meromorphic functions of F0 which is the fraction field of A, we get that d F0 ∧ 1 ⊗ M to induces an isomorphism of M-vector spaces (of dimension µ0 ) from Hvan (2 /d F0 ∧ d0 ) ⊗ M. It has been proved by M. Sebastiani (see [29, page 416]) that the A-module 2 /d F0 ∧ d0 is free of rank µ0 . Hence 2 /d F0 ∧ d0 + F0 2 is a Cvector space of dimension µ0 . If [α j ], j = 1, . . . , µ0 , is a basis of the µ0 -dimensional C-vector space 2 /d F0 ∧ d0 + F0 2 , we see that there is a family of 1-forms ω j 1 ⊗ M. such that ω j ∧ d F0 = α j is a basis of Hvan 5.2. Applications As a first application of the previous discussion, we get the following result observed by B. Malgrange∗ (see also [34] for the hyperelliptic case, and [41]). THEOREM 1 If F0 : (T ? R

= R2 , 0) → (R, 0) is a germ of analytic function and admits an isolated singularity at zero whose multiplicity is µ0 , hF0 i is of codimension µ0 ; we always have µ = µ0 . Proof It is clear, by using the isomorphism from 0 = E into 2 given by ϕ → ϕd x ∧ dξ , that we get an isomorphism between E /({E , F0 }+ F0 E ) and 2 /(d F0 ∧d0 + F0 2 ) which is a µ0 -dimensional vector space over C by Section 5.1. A simple proof of Theorem 1 in the quasi-homogeneous case is given in Section 8. Theorem 1 admits a very nice geometrical interpretation that we can derive from [34]. If χ is a germ of a canonical transformation near the origin, action integrals over small cycles are preserved. Hence any (uni)versal deformation should be able to reproduce the variations of the action integrals over the vanishing cycles. This is strongly consistent with the fact that µ is also the number of vanishing cycles, as shown in [31]. This is exactly the way things work in the quasi-homogeneous case, as shown in Section 9; we show there how to get the versal deformation theorem for quasi-homogeneous singularities. If the singularity is not quasi-homogeneous, E F0 + {E , F0 } is no more than the Jacobian ideal; indeed, K. Saito proved in [36] that (F ∈ J(F)) implies (F quasihomogeneous). In other words, there are deformations that are trivial as singularities ∗ This

was through an oral communication.

` YVES COLIN DE VERDI ERE

276

of functions, but not for the symplectic version. There is always a choice of a versal deformation which is valid for both problems: a pair of vector subspaces of the same codimension always admit a common supplementary subspace. For an example of a non-quasi-homogeneous singularity, we can take the singularity called Z 11 (µ = 11) in [4], which is given by Fa = x 3 ξ + ξ 5 + axξ 4 . Different values of a give nonequivalent singularities of functions but equivalent ideals. If F0 = 0 is a germ of a singular curve, we can associate to it a de Rham complex as in [25]: 0 → E → 1 /K → 0, where the nontrivial arrow is d and K is the set of 1-forms that vanish on the tangent vectors to the smooth stratum of F0 = 0:  K = α ∈ 1 ∃β ∈ 2 , α ∧ d F0 = F0 β . Then we define 1 1 0 Hde Rham (hF0 i) =  /(K + d ).

There is a subspace of the space of infinitesimal deformations which we can identify 1 1 with Hde Rham (hF0 i). If α ∈  is a germ of a 1-form, it gives a deformation of (F0 , ω0 ) defined by (F0 , ω0 + εdα). It is easy to check that the cohomology of [α] vanishes if and only if the deformation is trivial. Definition 8 The Tyurina number τ of F0 is defined by
τ = dim E / E F0 + J(F0 ) . The number τ is the dimension of the versal deformation of the ideal generated by F0 . It follows from the previously quoted result by Saito that τ = µ if and only if F0 is quasi-homogeneous. We can summarize the situation as follows (see also [41]). THEOREM 2 The following sequence of C-vector spaces is exact:

0→

0 0 1 → → → 0, d0 + K F0 0 + {F0 , 0 } F0 0 + J (F0 )

where the first nontrivial arrow is induced by α → d F0 ∧ α/d x ∧ dξ and the second is the canonical surjection. In particular, we have µ = τ + b1 .

SINGULAR LAGRANGIAN MANIFOLDS

277

This result is certainly not new, but we were unable to locate it in the literature. The meaning of the previous exact sequence is (deformations with hF0 i fixed ) → (deformations of (hF0 i, ω0 )) → (deformations of hF0 i). The exactness is easily checked from the definitions. 6. Normal forms THEOREM 3 Let (hF0 i, ω0 ) be a singular germ of a curve, and let ω be another germ of a symplectic form. If ω/ω0 > 0 and ω = ω0 + dα where the cohomology class of α vanishes, (hF0 i, ω) ∼ (hF0 i, ω0 ). Conversely, if (hF0 i, ωt ) ∼ (hF0 i, ω0 ) and ωt − ω0 = dαt , the cohomology class of αt vanishes. In particular, if F0 is quasi-homogeneous and ω/ω0 > 0, we have (hF0 i, ω) ∼ (hF0 i, ω0 ). Proof We consider the path of symplectic forms ωt = ω0 + t (ω − ω0 ) (0 ≤ t ≤ 1). We need to find a diffeomorphism ψ such that ψ preserves the curves F0 = 0 and ψ ? (ω) = ω0 . We can use the homotopy (Moser) trick, following [25, Theorem 1] : we try to find a family ψt of germs of diffeomorphisms associated to the time-dependent vector field d X t by X t (ψt (x)) = dt ψt (x), such that ? (1) ψt (ωt ) = ω0 , (2) the curve F0 = 0 is invariant by ψt . Condition (1) is satisfied if and only if d(ι(X t )ωt ) + ω − ω0 = 0, which, if dα = ω − ω0 , is implied by ι(X t )ωt = −(α − d f ) for any function f . Condition (2) is then satisfied as soon as α − d f vanishes on the vectors tangent to the smooth part of F0 = 0. This last condition can be fulfilled if and only if [α] = 0. Question 5

Does hF0 i, ω ∼ hF0 i, ω0 with ω/ω0 > 0 imply ω − ω0 = dα with [α] = 0? 7. Versal deformations: The formal case THEOREM 4 Let L0 = hF0 i (d = 1) be a germ of a singular Lagrangian manifold of codimenPµ sion µ in the sense of Definition 6, and let us denote by F0 + α=1 aα K α a versal P∞ k ∞ deformation of F0 . Let Lε = hFε i with Fε = k=0 ε Fk + O(ε ) be a smooth deformation of L0 = hF0 i.

` YVES COLIN DE VERDI ERE

278

Then there exists a smooth family of canonical transformations χε , a smooth invertible function E ε (x, ξ ), and smooth functions aα (ε) = O(ε) such that µ   X aα (ε)K α + O(ε∞ ). Fε ◦ χε = E ε F0 + α=1

Question 6 May we conjecture on the basis of the Morse case and of the proof that the formal series aα (ε) are uniquely defined? Proof We assume that Fε = F0 +

µ X

aα (ε)K α + εn Rn + O(εn+1 ).

α=1

Id +O(εn )

We need to find χε = vector field of X such that

= exp(εn Z )+ O(εn+1 ), where Z is the Hamiltonian

 µ X  X n Fε ◦ χε = (1 + ε E) F0 + aα (ε)K α + ε bα K α + O(εn+1 ). n

α=1

By identification of terms in

εn ,

we get the equation X {F0 , X } − E F0 = −Rn + bα K α ,

which can be solved in a fixed open set by the hypothesis of finite codimension. We assume that hF0 i is of codimension µ. Let hFε i be a smooth deformation of hF0 i. A basic question is whether there exist a smooth canonical deformation of the identity χε , a smooth deformation E ε of the function 1, and smooth functions aα (ε) = O(ε), such that   X Fε ◦ χε = E ε F0 + aα (ε)K α . (2) α

The transformations χε then move the deformation hFε i of hF0 i into the universal P one, hF0 + aα (ε)K α i. The condition of finite codimension allows us to solve the linearized problem, so it is natural to ask the following. Question 7 Does there exist in this context an implicit function theorem “`a la Mather”? The answer is yes for all simple singularities of curves in the holomorphic case (see Section 9.4).

SINGULAR LAGRANGIAN MANIFOLDS

279

8. The quasi-homogeneous case 8.1. Definitions We give the following. Definition 9 F = F(x, ξ ) is (a, b, N )-quasi-homogeneous, where a, b, and N are positive integers with a and b coprime, if F is a polynomial satisfying the identity F(t a x, t b ξ ) = t N F(x, ξ ). N this space of polynomials. We denote by Ea,b pa+qb

Any monomial x p ξ q is in Ea,b usual products is graded by

. The algebra R[[x, ξ ]] of formal series with the

R[[x, ξ ]] =

∞ M

N Ea,b .

N =0

Concerning Poisson brackets, we have l+m−(a+b)

l m , Ea,b } ⊂ Ea,b {Ea,b

.

N ) of finite codimension, we can choose If hFi is quasi-homogeneous (F ∈ Ea,b quasi-homogeneous K α , and for any k, X N +k k+a+b k Ea,b = {Ea,b , F} + Ea,b F+ RK α , N +k K α ∈Ea,b

where the last sum is finite. 8.2. Using Euler identity THEOREM 5 If F is a quasi-homogeneous isolated singularity of Milnor number µ, that is, if
dim E /J(F) = µ, then hFi is of codimension µ. More precisely, J(F) = E F + {E , F}. Proof Let us denote A = ∂ F/∂ x, B = ∂ F/∂ξ . We have, by Euler identity, a 0 x A + b0 ξ B = F

` YVES COLIN DE VERDI ERE

280

with a 0 = a/N and b0 = b/N . We want to solve {X, F} + Y F = λA + ν B, where λ, ν are given and X, Y ∈ E are unknown functions. We get, by replacing {X, F} with A ∂∂ξX − B ∂∂ Xx , A

∂ X

  ∂X  + a 0 xY + B − + b0 ξ Y = λA + ν B, ∂ξ ∂x

and it is now enough to solve ∂X = −ν + b0 ξ Y, ∂x

∂X = λ − a 0 xY. ∂ξ

The integrability condition is (a 0 + b0 )Y + b0 ξ

∂Y ∂Y ∂λ ∂ν + a0 x = + , ∂ξ ∂x ∂x ∂ξ

which admits a unique solution Y ; we first solve the inside formal series, then the inside flat functions. We can take for the U j ’s a basis of neighbourhoods star-shaped with respect to quasi-homogeneous dilatations.

9. Versal deformations for quasi-homogeneous singularities 9.1. A remarkable identity We have the following. 4 For any quasi-homogeneous singularity F0 with F0 (t a x, t b ξ ) = t N F0 (x, ξ ), we have the identity µ X
Nα = µ N − (a + b) , PROPOSITION

α=1

where (K α ) is a family of monomials defining a versal deformation and K α (t a x, t b ξ ) = t Nα K α (x, ξ ). Proof Following [4], we introduce the Poincar´e polynomial P(t) of the singularity as follows: µ X P(t) = t Nα . α=1

SINGULAR LAGRANGIAN MANIFOLDS

281

The following result is proved in [4, pages 166 – 168]: P(t) =

t N −a − 1 t N −b − 1 · b . ta − 1 t −1

It is clear that P(1) = µ, and hence µ= We see also that result.

P

N −a N −b · . a b

Nα = P 0 (1). By computing the derivative at t = 1, we get the

9.2. Nonvanishing of the Jacobian determinant of action integrals LEMMA 2 Let Fa (x, ξ ) ((x, ξ ) ∈ C2 , a ∈ Cµ ) be a versal deformation of a quasi-homogeneous singularity, and let γ j be a locally constant basis of the vanishing homology. Then R the Jacobian determinant J (a) of a → ( γ j (a) ξ d x) which is well defined outside the discriminant set (the set of a’s for which the curve Fa = 0 is singular) extends to Cµ as a nonvanishing holomorphic function. If we take the versal deformation generated by monomials, J is constant. As a corollary we get that there exists a canonical measure on the versal deformation (because the vanishing homology has a canonical Lebesgue measure). It would be nice to have a geometric definition of that measure. Proof •

We first check that ∂ ∂aα

•

•

Z γ (a)

ξ dx =

Z γ (a)

K α dt,

where dt is the time for the dynamics induced by the Hamiltonian Fa on the surface Fa = 0. We then prove, using the Picard-Lefschetz formula, that J is univalent. The Poincar´e group of the complement of the discriminant is generated by small loops around the stratum corresponding to 1 vanishing cycle, say, γ1 . Following such a loop adds to the lines of the Jacobian determinant a linear combination of the first one. J is bounded near the codimension 1 stratum of the discriminant. Hence J is holomorphic near the codimension 1 strata and, by Hartog’s theorem, everywhere.

` YVES COLIN DE VERDI ERE

282

1 J is quasi-homogeneous of degree zero:

CLAIM

J (t N −N1 a1 , . . . , t N −Nµ aµ ) = J (a1 , . . . , aµ ). Proof Fa (x, ξ ) gets multiplied by t N if x 7→ t a x, ξ 7→ t b ξ , aα 7→ t N −Nα aα . This implies that F = 0 is invariant under the latter transformation. Under this transformation the integral of ξ d x over γ (a), where the latter curve is determined by the condition that it be contained in F = 0, gets multiplied by t a+b . It follows that the derivative with respect to aα gets multiplied by t a+b−N +Nα , and therefore J gets multiplied by t k , P with k = µ(a + b − N ) + α Nα , if aα 7→ t N −Nα aα . Proposition 4 implies k = 0. If we choose the versal deformation so that K 1 = 1, J (a1 , 0, . . . , 0) is nonvanishing for a1 6= 0; it is a corollary of the discussion of Section 5.1 and of the explicit computation of the Jacobian. Because we have J (t N , 0, . . . , 0) = J (1, 0, . . . , 0), we see that J (0) 6= 0. This concludes the proof of Lemma 2. 9.3. Lifting isotopies P Let us give Fa (x, ξ ) = F0 (x, ξ ) + aα K α (x, ξ ), a (mini-)versal deformation of F0 where F0 admits an isolated singular point at the origin. We do not assume in this section that F0 is quasi-homogeneous. Let us denote the following: • z = (x, ξ ) ∈ C2 , • F(z, a) = Fa (z), • π : C2+µ → Cµ the canonical projection, • Z = {(z, a) | Fa (z) = 0}, • X a = {z | Fa (z) = 0}, • δ the critical set  δ = (z, a) z is a singular point of X a , • •

1 = π(δ) the discriminant set, 11 the set of a’s for which X a admits a unique singular point that is nondegenerate (a double point). The set 11 is a submanifold of codimension 1 in Cµ , whose closure is 1.

3 The critical set δ is smooth. LEMMA

SINGULAR LAGRANGIAN MANIFOLDS

283

This fact is well known and much more general. Here is a simple proof in our case. Proof If we can take K 1 = 1, K 2 = x, K 3 = ξ , then δ is a graph (a1 , a2 , a3 ) = G(a4 , . . . , aµ , x, ξ ). Otherwise, F0 is an Ak -singularity (ξ 2 ± x k+1 ) (there exists at least one derivative of F0 of order 2 nonvanishing at zero) and the result can easily be checked. We have the following. 4 P Let X = Aα (a) ∂a∂ α be a (germ of a) holomorphic vector field in Cµ which is tangent to 11 . There exists a holomorphic (germ of a) vector field X˜ on C2+µ tangent to Z which satisfies π? ( X˜ |Z ) = X . LEMMA

Proof The vector field X˜ should satisfy ∂ ∂ ∂ + U (z, a) + V (z, a) , ∂aα ∂x ∂ξ P and X˜ Fa vanishes on Z . In other words, X F = Aα K α belongs to the ideal J generated by F and its partial derivatives with respect to x and ξ . J is the ideal of the definition of the smooth manifold δ. Hence it is enough to prove that X F vanishes on δ. Let us fix a0 ∈ 11 , and let z 0 be the singular point of X a0 . We have π 0 (z 0 , a0 )(T(z 0 ,a0 ) Z ) = Ta0 11 ; this is because the map σ : a → z from 11 to δ, where z is the Morse singular point of X a , is a section of π over 11 . Let W0 ∈ T(z 0 ,a0 ) Z be such that π 0 (z 0 , a0 )(W ) = X (a0 ). We have W0 (F) = X (a0 )(F) because the derivatives of F with respect to z vanish at that point. We deduce X (a0 )F = 0 because W0 is tangent to Z . It follows that X F, vanishing on δ, the closure of σ (11 ), belongs to J . X˜ =

X

Aα (a)

We need the following. COROLLARY 1 If a → ϕt (a) is a smooth isotopy that preserves 11 , it can be lifted to a smooth isotopy 8t (a, z) on C2+µ which preserves Z . In other words, we have 8t (z, a) = (ψta (z), ϕt (a)), where ψta is a (germ of) diffeomorphism of C2 which maps X a onto X ϕt (a) .

` YVES COLIN DE VERDI ERE

284

Proof The corollary is proved by just integrating the lift X˜ t , built using Lemma 4, of the d time-dependent vector field X t (ϕt (a)) = dt ϕt (a). 9.4. Versal deformation theorem in the holomorphic case We prove the versal deformation theorem for all quasi-homogeneous singularities. Using the strategy of Pham in [34], we can prove the following. 6 P Let hF0 i be a quasi-homogeneous singularity with Fa = F0 + aα K α (K α monomials) as a versal deformation. Let hFt i be any analytic deformation of hF0 i. There exists an analytic family of germs of canonical diffeomorphisms χt such that hFt ◦ χt i = hFa(t) i; that is,   X Ft ◦ χt = E t F0 + aα (t)K α , THEOREM

where the functions aα (t), E t are analytic. Remark. The previous result could be extended, with the same proof, to every isolated singularity if we were able to prove the nonvanishing of an appropriate Jacobian determinant. Proof We give the proof for A2 (the cusp); it is then trivial to see how to extend the proof to the general case. Using Moser’s method, the idea is to fit the action integrals. The details run as follows. • We can assume, using the versal deformation theorem (see [4]), that our deformation is embedded into the deformation (Fa = F0 + a1 x + a2 , ωt ), where ωt = ω0 + O(t). We choose λt such that dλt = ωt − ω0 , and we assume that λt = O(|t|). • Let 1 = {4a13 +27a22 = 0} be the discriminant set. We want to define a smooth family of holomorphic diffeomorphisms (an isotopy) a → ϕt (a) = a 0 such that ϕ0 = Id, and for all cycles γ j of X a = {Fa = 0}, we have Z Z ξ dx = (ξ d x + λt ). (3) γ j (a)

γ j (a 0 )

This implicit equation can be uniquely solved for t small enough outside 1 R because the Jacobian determinant of a → ( γ j (a) ξ d x) j=1,2 is a nonzero constant (see Lemma 2).

SINGULAR LAGRANGIAN MANIFOLDS

•

•

•

285

Near the stratum of the regular part 11 of the discriminant set where the vanR R R R ishing cycle is γ1 , the integrals γ1 and γ2 ± γ1 log γ1 are univalent and holomorphic, thanks to the Picard-Lefschetz formula, and the Jacobian determinant is the same, so we can also solve equation (3). Now that we have solved equation (3) outside a set of codimension 2, we can solve it everywhere using the fact that holomorphic functions have no singularities of codimension greater than or equal to 2 (Hartog’s theorem). Using Corollary 1, we get a diffeomorphism 8t that lifts ϕt . We have then Z Z ξ dx = (ψta )? (ξ d x + λt ). γ j (a)

•

•

γ j (a)

We put ω0 = dξ ∧ d x and ωt0 = (ψta )? (ωt ). The deformations (Fa 0 , ωt ) and (Fa , ωt0 ) are clearly equivalent. The difference of the 2 symplectic forms ωt0 and ω0 is dβta , where the integrals of βta over all vanishing cycles of all X a ’s vanish. It remains now to find f a,t (x, ξ ) whose differential on X a is βta . We build f a,t so that f (a1 ,b),t = ga1 ,t is independent of b. The differential of ga1 ,t restricted to X a1 ,b where b varies is given by the restriction of βta1 ,b to X a1 ,b . We get ga1 ,t by integrating from a point m a1 ,b ∈ Z a1 ,b ∩ {kzk = 1} which can be chosen to be an analytic function of (a1 , b) of the differential forms βta1 ,b . The smoothness of g outside 1 is clear. Moreover, g is holomorphic outside 1 and bounded near 11 , and hence is holomorphic everywhere. We can then apply Moser’s method.

10. Semiclassics In this section, we quantize everything in order to get semiclassical objects. 10.1. Semiclassical normal forms THEOREM 7 Let hH0 i be of finite codimension µ with a (classical) real versal deformation genb be a pseudodifferential operator on R whose erated by K α , α = 1, . . . , µ. Let H b principal symbol is H0 . There exist then some elliptic pseudodifferential operators U b and formal series aα (h) = O(h) such that we have microlocally near zero and V X bH bV b= c cα + O(h ∞ ), U H0 + aα (h) K α

b is the Weyl quantization of Q. If H b is self-adjoint, we can choose U b and V b where Q so that the aα ’s are real valued. The proof by induction on the powers of h is similar to the proof of Theorem 4.

` YVES COLIN DE VERDI ERE

286

10.2. Mixed case We consider now a smooth family c Hε of semiclassical Hamiltonians and denote by H0 the principal symbol of c H0 . We assume that hH0 i is of finite codimension µ. The following result is an extension of Theorem 4 (h = 0) and Theorem 7 (ε = 0). 8 cε and Vbε and formal series There exist elliptic pseudodifferential operators U aα (ε, h) = O(|h| + |ε|) such that X cε c cα + O(ε∞ + h ∞ ). U Hε Vbε = c H0 + aα (ε, h) K THEOREM

α

The proof is by induction on the powers of h and for each power of h by induction on the powers of ε. 10.3. The case of quasi-homogeneous singularities In the holomorphic quasi-homogeneous case, using the tools of Section 9.4, we get a much better result. Definition 10
P j bE = OpW We say that H h H j (E; x, ξ ) is an analytic family of pseudodifferential operators near zero if, for all indices j, H j (E, x, ξ ) extends to a holomorphic function in some complex neighbourhood  of zero independent of j. THEOREM 9 bE is an analytic family of pseudodifferential operators of order zero such that If H H0 (0; x, ξ ) is a quasi-homogeneous singularity, there exists, for E small enough, an analytic family of unitary Fourier integral operators U E , an analytic family of elliptic pseudodifferential operators FE , and symbols (analytic with regard to E) aα (E, h) such that we have, microlocally near zero,   X bE U E = FE ◦ H b0 + bα + O(h ∞ ). U E? H aα (E, h) K

Proof Proceeding by induction on the powers of h, we get the following equation to solve, where X (E; x, ξ ), Y (E; x, ξ ), and cα (E) are the unknown functions: n

H0 +

X

o   X aα (E)K α , X + Y H0 + aα (E)K α = R(E; x, ξ ) −

X

cα (E)K α (x, ξ ).

SINGULAR LAGRANGIAN MANIFOLDS

287

P This equation expresses the fact that on the Riemann surface H0 + aα (E)K α = P 0, R(E; x, ξ ) − cα (E)K α (x, ξ ) is the derivative with respect to the time of the function X . We first need to choose cα (E) so that the integrals Z   X R(E; x, ξ ) − cα (E)K α (x, ξ ) dt γ j (E)

all vanish. This is possible outside the discriminant set because of the nonvanishing R of the determinant γ j (E) K α dt (see Lemma 2). The solution is bounded near the discriminant and hence can be extended to a holomorphic function. The proof is then finished using the same arguments as in the proof of Theorem 6. 11. Singular Bohr-Sommerfeld rules: The general scheme From the local model and the WKB solutions, we define the scattering matrices and singular holonomies. We show how one can take the principal part of the regular holonomies in order to get the singular holonomies. We can then derive the BohrSommerfeld rules using the same combinatorial recipe as in [14] (maximal trees, . . .). 11.1. The context bE is a pseudodifferential operator of order zero on the real line, and We assume that H we denote by H E its principal symbol. H is supposed to be real valued, and we assume that the energy surface Z = H0−1 (0) admits only finite codimension singularities z j , j = 1, . . . , N , with normal forms bVbj = H cj + cj H U

µj X

∞ a j,α (E, h) Kd j,α + O(h ),

(4)

α=1

where a j,α (E, h) are symbols in h and Kd j,α are Weyl quantizations of the real versal deformation. 11.2. Local models and scattering matrices In this section we want to describe the solutions of the local model that is mapped on our problem near the singular point z j . We omit the index j in this section. We fix a neighbourhood  of zero in the (y, η) symplectic plane. We denote by P ca its (Weyl-) Ha = H0 + aα K α the versal deformation of the model and by H 0 quantized version. We denote by γl , l = 1, . . . , L = 2L (L ≥ 2) the real branches of the germ Z a = Ha−1 (0). We choose to orient the γl ’s according to the dynamics of Ha . There are now L 0 ingoing and L 0 outgoing branches. We choose open sets l ⊂  with empty mutual intersections and such that l ∩ γl is a nonempty connected arc

` YVES COLIN DE VERDI ERE

288

2

γ3

γ2

Za

 γ1

γ4

1

γ5

γ6

Figure 2. The model problem: L = 6

(see Figure 2). We assume that a is small enough so that l ∩ Z a with Z a = Ha−1 (0) is also a nonempty connected arc. We are looking for the following equation:   X c cα u = O(h ∞ ), H0 + aα K (5) where u is a microfunction in . It is in general not difficult to prove that the space of microfunction solutions of equation (5) in  is a free module of rank L 0 = L/2 over the moderate growth functions of h. We choose microlocal solutions u l (a) of equation (5) inside l smoothly dependent of a of the form (in case of no caustics) u l (a; x) ∼

∞ X

 ck,l (a, x)h k ei Sl (a;x)/ h

(6)

k=0

with ck,l and Sl smoothly depending on a. Any solution u of equation (5) in  restricts to xl u l (a) in l . Given (xl ) = (xin , xout ), we can express the condition that xl u l (a) are the restrictions to l of some solution u of equation (5) by a matrix xout = S (a, h)xin ,

(7)

where S (a, h) is called the scattering matrix. 11.2.1. Unitarity ba is formally self-adjoint. Let us choose 5 a pseudoWe assume that the operator H differential operator of order zero compactly supported in  and equal to Id near the origin. More precisely, we assume that  [ Z a ∩ 5(Id −5) 6= 0 ⊂ (l ∩ Z a ). l

SINGULAR LAGRANGIAN MANIFOLDS

289

We define the following inner products on microfunctions in : Ja (u, v) =

i ba ]u, vi. h[5, H h

It is clear that ba ), J (u, v) = O(h ∞ ); (1) if u, v ∈ ker( H P (2) if u |l = xl u l and v|l = yl u l , we have J (u, v) = l xl y¯l J (u l , u l ); (3) if the principal symbol of u l is |dt|1/2 , we have J (u l , u l ) = ±1+ O(h), where we have a + sign if the arc γl is ingoing and a − sign if it is outgoing. From that we deduce that S (a, h) is unitary (with maybe some domain). 11.3. Singular holonomies Let γ0 be a cycle of Z 0 ; we want to define the singular holonomy (of d H E ) along γ and compute it. For simplicity we assume that there exists only one singular point z 1 in γ0 at which we have a normal form given by equation (4). We can therefore omit the index j. We first cover the cycle γ0 by open sets U1 , . . . , Un (see Figure 3) such that we can find WKB solutions v j of c H0 v = O(h ∞ ) inside U j , points ζ j = (a j , b j ) ∈ U j ∩ U j+1 , and such that the  j ’s covering the singular point z 0 ( j = 1, n) are the image by the canonical transformation χ of some open sets l , l = 1, 2, introduced in the previous section. We choose v1 = Vb1 u 1 and vn = Vb1 (u 2 ). We define then then the singular holonomy HolS( c H0 , γ0 ) by HolS( c H0 , γ0 ) =

n−1 Y j=1

v j (a j ) . v j+1 (a j )

(8)

It P is clear from the theory of the WKB-Maslov ansatz that HolS( c H0 , γ0 ) = i( ∞ Bk h k ) k=−1 e , so that we go to some log scale and put LHolS = −i log HolS ∼

∞ X

Bk h k .

k=−1

It is easily checked that singular holonomies are independent of all choices (including χ and the associated Fourier integral operators) except for the chosen WKB solutions u l of the model problem. As we see, singular holonomies and scattering matrices are enough to derive Bohr-Sommerfeld rules. 11.4. Regularization b such that H −1 (0) = Z E is smooth We now choose a deformation d H E , E ≥ 0, of H E and choose a cycle γ E of Z E such that γ E → γ0 as E → 0+ . The goal is to derive b, γ0 ) as a regularization of the usual holonomy (the log of) LHol(d LHolS( H HE , γ E ) ∼

` YVES COLIN DE VERDI ERE

290

4 (E)

n

3

z1 ζ4 ζ1

z1

ζ3

γE ζ2 γ0

1

2 Figure 3. Singular holonomy

P∞

Ak (E)h k . In general, the Ak ’s are divergent as E → 0+ , but we can substract the divergent part using the scattering matrix. More precisely, assume E > 0. We have then vn = s1,n (E, h)v1 , where s1,n is the corresponding entry of the local scattering matrix. We deduce the following: k=−1

LHol(E, h) = LHolS(E, h) +

1 log s1,n (E, h). i

For fixed E > 0, we then have k Ak (E) = Bk (E) + σ1,n (E),

where

∞ X 1 log s1,n (E, h) ∼ σ1,n (E)h k . i k=−1

We get in that way
k Bk (0) = lim Ak (E) − σ1,n (E) . E→0+

11.5. Singular Bohr-Sommerfeld rules Once the singular holonomies are defined, the Bohr-Sommerfeld rules follow the same combinatorial picture as in [14]. 12. The cusp The saddle-node bifurcation occurs generically for a 1-dimensional system depending on some extra parameter; it is the generic way to change the number of critical points for a Morse function.

SINGULAR LAGRANGIAN MANIFOLDS

291

Definition 11 We say that the planar curve L = hH i admits at z 0 a nondegenerate cusp if z 0 is a degenerate (non-Morse) critical point of H such that H 00 (z 0 ) is of rank 1 and the polynomial of degree 3 in the Taylor expansion does not vanish on the kernel of H 00 (z 0 ). 12.1. Classics THEOREM 10 Let H be a Hamiltonian such that hH i admits at z 0 a nondegenerate cusp; there exist a canonical transformation χ and a smooth function E nonvanishing at z 0 such that H ◦ χ = E H0 with H0 = ξ 2 + x 3 : hH ◦ χ i = H0 . By Theorem 3, it is enough to know that H and ξ 2 + x 3 are equivalent germs. This result can be proved easily as follows. First apply the Morse lemma; we get ξ 2 + f (x), where the third derivative of f does not vanish (see [4, Chapter 2]). 12.2. Semiclassics Let c Ht u = 0 be an analytic family of semiclassical equations such that the principal symbol H0 of c H0 vanishes at z 0 with a nondegenerate cusp. Using Theorems 6, 8, and 10, we get the following pseudodifferential equation as a microlocal normal form:
−h 2 u 00 + x 3 + a(t, h)x + b(t, h) u = O(h ∞ ), P P∞ j j where a ∼ ∞ j=0 a j (t)h and b ∼ j=0 b j (t)h are formal series in h. 12.3. Computation of the first coefficients a1,0 and b1,0 Let us start with F0 having a cusp at zero. By a rotation, we can assume that F0 = Aξ 2 + Bx 3 + O(7), where f = O(N ) means F(t 3 ξ, t 2 x) = O(t N ). By a canonical diagonal linear transformation, we get
F0 = (A3 B 2 )1/5 ξ 2 + x 3 + αx 2 ξ + βxξ 2 + γ x 4 + O(9) , and then, removing the constant prefactor,  α2  2 α 3  F0 = ξ 2 + x + ξ + β − xξ + γ x 4 + O(9), 3 3 and putting ξ1 = ξ, x1 = x + (α/3)ξ ,  α2  2 F0 = ξ12 + x13 + β − x1 ξ1 + γ x14 + O(9). 3

` YVES COLIN DE VERDI ERE

292

We want to find χ so that F0 ◦ χ = (1 + ex1 )(ξ12 + x13 ) + O(9). We easily compute 4 3 x(ξ 2 + x 3 ) + {S, ξ 2 + x 3 }, x 4 = x(ξ 2 + x 3 ) + {S 0 , ξ 2 + x 3 }, 7 7

xξ 2 =

and we get that way e=

1 (3β + 4γ − α 2 ). 7

We now have  K ◦χ  (F0 + t K ) ◦ χ = (1 + ex1 ) ξ12 + x13 + t + O(9), 1 + ex1 and by projecting the deformation onto the versal deformation, we get   (F0 + t K ) ◦ χt = E t (x1 , ξ1 ) ξ12 + x13 + t (a1,0 x1 + b1,0 ) + O(t 2 ). We put k0 = K (0), k1 = ∂x1 K (0), and we get a1,0 = k1 − ek0 ,

b1,0 = k0 .

The same formulae hold for a0,1 and b0,1 by replacing K with the subprincipal symbol b0 . of H 12.4. The model problem b Let Pv(y) = −v 00 (y) + (y 3 + Ay + B)v(y) with A, B ∈ R. We may define the reflexb = 0 admits 2 exact ion coefficient R(A, B) in the following way. The equation Pv solutions v± (y), smoothly depending on A and B, which admit WKB expansions at infinity (v− (y) = v+ (y)) and, as y → −∞, v+ (y) = |y|−3/4 ei((2/5)|y|

5/2 +A|y|1/2 )

∞   X 1+ aα (A, B)|y|−α/2 + O(|y|−∞ ). α=1

The existence of solutions with a given asymptotic expansion is a classical fact. They are clearly unique. (For a general approach concerning asymptotic solutions, see [6], [35], [40].) There exists a unique function R(A, B) (of modulus 1), called the reflexion coefficient or scattering matrix, such that v = v− + R(A, B)v+ ∈ L 2 ([0, +∞[, dy). This function R(A, B) is the special function of our problem. It can be related to Stokes multipliers. Problem 2 Describe as much as possible the function R(A, B).

SINGULAR LAGRANGIAN MANIFOLDS

293

b

E

C

H

Figure 4. Bifurcation diagram of the cusp

a

` YVES COLIN DE VERDI ERE

294

12.5. The semiclassical bifurcation 12.5.1. The scattering matrix We choose exact solutions u ±,a,b of equation −h 2 u 00 + (x 3 + ax + b)u = O

(9)

(with a and b real valued) which admit the following WKB expansions: u ±,a,b (x, h) = e

i Sa,b (x)/ h

∞ X

 a j (x; a, b)h j + O(h ∞ )

j=0

P j normalized by Sa,b (−1) = 0 and ∞ j=0 a j (−1; a, b)h = 1. We obtain that way the semiclassical scattering matrix σ (a, b; h), well defined modulo O(h ∞ ), by asking that u +,a,b + σ (a, b; h)u −,a,b extend to an admissible function (see the classical pictures in Figure 4). 12.5.2. Renormalization Let us start with the semiclassical model problem given by equation (9) and assume that a and b can be h dependent. We denote ka, bk = (|a|3 + b2 )5/12 , and we measure the distance to the bifurcation using τ defined by ka, bk = hτ = η. We can now use the renormalisation x = η2/5 y, which gives −τ −2 v 00y 2 + (y 3 + Ay + B)v = 0 with a = Aη4/5 , b = Bη6/5 . Now A and B are of order 1. We have 3 domains: (1) the domain where τ is bounded (with respect to h), where the bifurcation really takes place and there is no further asymptotics; (2) the log domain where 1  τ = O(| log h|), where we can use the semiclassical asymptotics with regard to τ including the tunneling effect which is not O(h ∞ ); (3) the domain where τ  | log h|, where we can apply usual formulae without looking at the bifurcation problem: the semiclassical spectrum splits into 2 parts, one associated to the real vanishing circle, the other to the big closed cycle.

SINGULAR LAGRANGIAN MANIFOLDS

295

12.5.3. The bifurcation domain In this domain (ka, bk = O(h)), τ is bounded. If we use a = Ah 4/5 , b = Bh 6/5 , the renormalized equation is −v 00 + (x 3 + Ax + B)v = O

(10)

where A and B are bounded. In this domain, we have the following relationship between R and σ : σ (a, b; h) = R(A, B)e

−(i/ h)((4/5)+2Ah 4/5 )



1+

∞ X

 γα (A, B)h α/5 + O(h ∞ ) (11)

α=1

with A = ah −4/5 , B = bh −6/5 , and the γα ’s can be computed from the aα ’s. 12.5.4. The log domain In this domain we can compute the τ semiclassical solution using the tunneling effect (see [21], [18]). 12.6. Bohr-Sommerfeld rules From the previous sections we can compute the singular holonomy using the asymptotic behaviour of σ (a, b; h) for (a, b) nonzero and h → 0. We can then derive the Bohr-Sommerfeld rules from R(A, B) using equation (11). Acknowledgments. The papers of Eric Delabaere and Fr´ed´eric Pham, as well as discussions with them, were important sources of inspiration. Emmanuel Ferrand provided me with a basic list of references on singular Lagrangian manifolds. I thank also Bernard Malgrange for remarks on preliminary versions, discussions, and “Theorem 1,” and San V˜u Ngo.c for carefully reading a preliminary version. Pertinent remarks of the referees were helpful in order to make the paper correct and legible. References [1] [2] [3]

[4]

V. I. ARNOLD, Lagrangian manifolds with singularities, asymptotic rays and the open

swallowtail, Funct. Anal. Appl. 15, no. 4 (1981), 235 – 246. MR 83c:58011 267 , Mathematical Methods of Classical Mechanics, 2d ed., Grad. Texts in Math. 60, Springer, New York, 1989. MR 90c:58046 , “First steps of local symplectic algebra” in Differential Topology, Infinite-Dimensional Lie Algebras, and Applications: D. B. Fuchs’ 60th Anniversary Collection, Amer. Math. Soc. Transl. Ser. 2 194, Amer. Math. Soc., Providence, 1999, 1 – 8. MR 2001h:58055 ´ and A. N. VARCHENKO, Singularit´es des V. I. ARNOLD, S. M. GOUSSEIN-ZADE, applications diff´erentiables, I, Mir, Moscow, 1986. 265, 276, 280, 281, 284, 291

296

` YVES COLIN DE VERDI ERE

[5]

V. I. ARNOLD, V. A. VASILEV, V. V. GORYUNOV, and O. V. LYASHKO, Dynamical

[6]

W. BALSER, Formal Power Series and Linear Systems of Meromorphic Ordinary

Systems, VI, Encyclopedia Math. Sci. 6, Springer, Berlin, 1993. MR 94b:58018

[7] [8]

[9] [10] [11]

[12]

[13]

[14] [15] [16] [17]

[18] [19] [20] [21] [22] [23]

Differential Equations, Universitext, Springer, New York, 2000. MR 2001f:34169 292 S. BATES and A. WEINSTEIN, Lectures on the Geometry of Quantization, Berkeley Math. Lect. Notes 8, Amer. Math. Soc., Providence, 1997. MR 2002f:53151 P. J. BRAAM and J. J. DUISTERMAAT [H. DUISTERMAAT], Normal forms of real symmetric systems with multiplicity, Indag. Math. (N.S.) 4 (1993), 407 – 421. MR 94k:58146 270 E. CALICETI, S. GRAFFI, and M. MAIOLI, Perturbation of odd anharmonic oscillators, Comm. Math. Phys. 75 (1980), 51 – 66. MR 82d:81030 265 E. CALICETI and M. MAIOLI, Odd anharmonic oscillators and shape resonances, Ann. Inst. H. Poincar´e Sect. A (N. S.) 38 (1983), 175 – 186. MR 85e:81027 265 ` Y. COLIN DE VERDIERE, M. LOMBARDI, and J. POLLET, The microlocal Landau-Zener formula, Ann. Inst. H. Poincar´e Phys. Th´eor. 71 (1999), 95 – 127. MR 2000j:81060 264 ´ ` Y. COLIN DE VERDIERE and B. PARISSE, Equilibre instable en r´egime semi-classique, I: Concentration microlocale, Comm. Partial Differential Equations 19 (1994), 1535 – 1563. MR 96b:58112 264 ´ , Equilibre instable en r´egime semi-classique, II: Conditions de Bohr-Sommerfeld, Ann. Inst. H. Poincar´e Phys. Th´eor. 61 (1994), 347 – 367. MR 97a:81041 264 , Singular Bohr-Sommerfeld rules, Comm. Math. Phys. 205 (1999), 459 – 500. MR 2000k:81088 264, 266, 287, 290 ` Y. COLIN DE VERDIERE and SAN VU˜ NGO.C, Singular Bohr-Sommerfeld rules for 2D ´ integrable systems, to appear in Ann. Sci. Ecole Norm. Sup. (4). 264 ` Y. COLIN DE VERDIERE and J. VEY, Le lemme de Morse isochore, Topology 18 (1979), 283 – 293. MR 80k:57059 269, 271 E. DELABAERE, H. DILLINGER, and F. PHAM, R´esurgence de Voros et p´eriodes des courbes hyperelliptiques, Ann. Institut Fourier (Grenoble) 43 (1993), 163 – 199. MR 94i:34115 , Exact semiclassical expansions for one-dimensional quantum oscillators, J. Math. Phys. 38 (1997), 6126 – 6184. MR 99c:81041 295 E. DELABAERE and F. PHAM, Unfolding the quartic oscillator, Ann. Physics 261 (1997), 180 – 218. MR 99k:81065 265 E. DELABAERE and D. T. TRINH, Spectral analysis of the complex cubic oscillator, J. Phys. A 33 (2000), 8771 – 8796. MR 2001k:81064 265 R. B. DINGLE, Asymptotic Expansions: Their Derivation and Interpretation, Academic Press, London, 1973. MR 58:17673 295 J. J. DUISTERMAAT [H. DUISTERMAAT], Fourier Integral Operators, Progr. Math. 130, Birkh¨auser, Boston, 1996. MR 96m:58245 268 L. H. ELIASSON, Normal forms for Hamiltonian systems with Poisson commuting integrals: Elliptic case, Comment. Math. Helv. 65 (1990), 4 – 35. MR 91d:58223

SINGULAR LAGRANGIAN MANIFOLDS

297

271 [24]

[25] [26] [27]

[28]

[29] [30] [31] [32]

[33]

[34]

[35] [36] [37]

[38]

[39]

A. T. FOMENKO, “Topological classification of all integrable Hamiltonian differential

equations of general type with two degrees of freedom” in The Geometry of Hamiltonian Systems (Berkeley, 1989), Math. Sci. Res. Inst. Publ. 22, Springer, New York, 1991, 131 – 339. MR 93e:58081 273 A. B. GIVENTAL’, Singular Lagrangian manifolds and their Lagrangian maps, J. Soviet Math. 52 (1990), 3246 – 3278. MR 91g:58077 276, 277 ¨ L. HORMANDER , Fourier integral operators, I, Acta Math. 127 (1971), 79 – 183. MR 52:9299 268 , The Analysis of Linear Partial Differential Operators, III: Pseudodifferential Operators, Grundlehren Math. Wiss. 274, Springer, Berlin, 1985. MR 87d:35002a , The Analysis of Linear Partial Differential Operators, IV: Fourier Integral Operators, Grundlehren Math. Wiss. 275, Springer, Berlin, 1985. MR 87d:35002b ´ B. MALGRANGE, Int´egrales asymptotiques et monodromie, Ann. Sci. Ecole Norm. Sup. (4) 7 (1974), 405 – 430. MR 51:8459 274, 275 V. P. MASLOV, Th´eorie des perturbations et m´ethodes asymptotiques, with appendices by V. I. Arnold and V. S. Busalaev, Dunod, Paris, 1972. 268 J. MILNOR, Singular Points of Complex Hypersurfaces, Ann. of Math. Stud. 61, Princeton Univ. Press, Princeton, 1968. MR 39:969 274, 275 NGUYEN HU’U DU’C and F. PHAM, Germes de configurations legendriennes stables et fonctions d’Airy-Weber g´en´eralis´ees, Ann. Inst. Fourier (Grenoble) 41 (1991), 905 – 936. MR 93e:58019 NGUYEN TIEN ZUNG [ZUNG, NGUYEN TIEN], A note on degenerate corank-one singularities of integrable Hamiltonian systems, Comment. Math. Helv. 75 (2000), 271 – 283. MR 2002e:37093 273 F. PHAM, “Multiple turning points in exact WKB analysis (variations on a theme of Stokes)” in Toward the Exact WKB Analysis of Differential Equations, Linear or Non-Linear (Kyoto, 1998), Kyoto Univ. Press, Kyoto, 2000, 71 – 85. MR 2002f:34222 264, 275, 284 J.-P. RAMIS, S´eries divergentes et th´eories asymptotiques, Bull. Soc. Math. France 121 (1993), Panor. Synth., supplement. MR 95h:34074 292 K. SAITO, Quasihomogene isolierte Singularit¨aten von Hyperfl¨achen, Invent. Math. 14 (1971), 123 – 142. MR 45:3767 275 ˜ NGO.C [VU ˜ NGO.C, SAN], Bohr-Sommerfeld conditions for integrable systems SAN VU with critical manifolds of focus-focus type, Comm. Pure Appl. Math. 53 (2000), 143 – 217. MR 2001m:81081 264 , Formes normales semi-classiques des syst`emes compl`etement int´egrables au voisinage d’un point critique de l’application moment, Asymptot. Anal. 24 (2000), 319 – 342. MR 2001m:58059 271 R. SCHOEN and J. WOLFSON, Minimizing area among Lagrangian surfaces: the mapping problem, J. Differential Geom. 58 (2001), 1 – 86. CMP 1 895 348 269

298

[40]

[41] [42] [43]

` YVES COLIN DE VERDI ERE

Y. SIBUYA, Linear Differential Equations in the Complex Domain: Problems of

Analytic Continuation, Transl. Math. Momogr. 82, Amer. Math. Soc., Providence, 1990. MR 92a:34010 292 D. VAN STRATEN and C. SEVENHECK, Deformation of singular Lagrangian subvarieties, preprint, arXiv:math.AG/0002083 265, 269, 275, 276 A. VOROS, The return of the quartic oscillator: The complex WKB method, Ann. Inst. H. Poincar´e Sect. A (N. S.) 39 (1983), 211 – 338. MR 86m:81051 , Exact resolution method for general 1D polynomial Schr¨odinger equation, J. Phys. A 32 (1999), 5993 – 6007. MR 2000k:81071 265

Institut Fourier, Unit´e Mixte de Recherche 5582 du Centre National de la Recherche Scientifique, Universit´e de Grenoble I, BP 74, 38402 Saint Martin d’H`eres CEDEX, France; [email protected]

NONSYMMETRIC MACDONALD POLYNOMIALS AND DEMAZURE CHARACTERS BOGDAN ION

Abstract We establish a connection between a specialization of the nonsymmetric Macdonald polynomials and the Demazure characters of the corresponding affine Kac-Moody algebra. This allows us to obtain a representation-theoretical interpretation of the coefficients of the expansion of the specialized symmetric Macdonald polynomials in the basis formed by the irreducible characters of the associated finite Lie algebra. Introduction Generalizing the characters of compact simple Lie groups, I. Macdonald associated to each irreducible root system a family of orthogonal polynomials Pλ (q, t) indexed by antidominant weights and invariant under the action of the Weyl group. These polynomials depend rationally on parameters q and t = (ts , tl ) and, for particular values of these parameters, reduce to familiar objects in representation theory. (1) When q = ts = tl , they are equal to χλ , the Weyl characters of the corresponding root system. (In particular, they are independent of q.) (2) When q = 0, they are the polynomials that give the values of zonal spherical functions on a semisimple p-adic Lie group relative to a maximal compact subgroup. (3) When ts = q ks , tl = q kl , and q tends to 1, they are the polynomials that give the values of zonal spherical functions on a real symmetric space G/K which arise from finite-dimensional spherical representations of G. Here ks , kl are the multiplicities of the short, respectively, long, restricted roots. The nonsymmetric Macdonald polynomials E λ (q, t) (indexed this time by the entire weight lattice) were first introduced by E. Opdam [O] in the differential setting and then by I. Cherednik [C2] in full generality. Unlike the symmetric polynomials, their representation-theoretical meaning is still unexplored. At present time their main importance consists of the fact that they form the common spectrum of a family of commuting operators (the Cherednik operators) which play a preponderant role in the DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 2, Received 8 May 2001. Revision received 19 November 2001. 2000 Mathematics Subject Classification. Primary 33D52; Secondary 17B10, 17B67, 20C08, 33D80.

299

300

BOGDAN ION

representation theory of affine Hecke algebras and related harmonic analysis. It became clear, especially from the work of Cherednik, that we can in fact construct such families of polynomials for every irreducible affine root system. From this point of view, the objects studied by Cherednik [C1], [C2], [C3] are the polynomials attached to reduced twisted affine root systems, and the Koornwinder polynomials, studied by S. Sahi [S2], [S4], are the polynomials attached to nonreduced affine root systems. This paper was inspired by the result of Y. Sanderson [Sa], who established a connection between a specialized version of the nonsymmetric Macdonald polynomials ˜ (E λ (q, ∞) in our notation) and the characters of certain Demazure modules E wλ (λ) of the irreducible affine Lie algebra (see Section 1 for the definitions of the ingredients) in the case of an irreducible root system of type An . Extrapolating from [Sa], we establish here the same connection for all irreducible affine root systems for which the affine simple root is short. This condition identifies precisely the polynomials studied by Cherednik and Sahi. The proofs rely heavily on the method of intertwiners in double affine Hecke algebras. THEOREM 1 For an affine root system as above and any weight λ, we have
˜ ˜ . E λ (q, ∞) = q (30 ,wλ hλi) χ E wλ (λ) (1)

(1)

(1)

(1)

The remaining cases, Bn , Cn , F4 , and G 2 , exhibit some special features. For example, the formula of the affine intertwiner as an element of the double affine Hecke algebra takes a different form (see [I]). Computations suggest that the action on the weight lattice of the degeneration of this affine intertwiner at t = ∞ does not equal the action of the affine Demazure operator but equals that of a different action with similar properties. The connection between nonsymmetric Macdonald polynomials and Demazure characters allows a representation-theoretical interpretation of the coefficients of the expansion of the symmetric polynomials in the basis formed by the irreducible characters of the associated finite Lie algebra. Our second result is the following. THEOREM 2 For an affine root system as above and any antidominant weight λ, the symmetric polynomial Pλ (q, ∞) can be written as a sum X Pλ (q, ∞) = dλµ (q)χµ , µ≤λ

where dλµ (q) is a polynomial in q −1 with positive integer coefficients.

NONSYMMETRIC MACDONALD POLYNOMIALS

301

Let us mention that in the An -case, as explained in [Sa], the positivity of the above coefficients is closely related to the positivity of the Kostka-Foulkes polynomials via the duality of the two variable Kostka functions. Another consequence of the Theorem 1 is the following. 3 For an affine root system as above and any weight λ, we have
E λ (∞, ∞) = χ E˚ w˚ λ w◦ (λ+ ) . THEOREM

This relates the specialization of the nonsymmetric Macdonald polynomials E λ (∞, ∞) = lim lim E λ (q, t) q→∞ t→∞

to the Demazure characters of the finite irreducible Lie algebras. The order in which we compute the above limits seems to be irrelevant.

1. Preliminaries 1.1. The affine Weyl group For the most part we adhere to the notation in [Ka]. Let A = (ai j )0≤i, j≤n be an irreducible affine Cartan matrix, let S(A) be the Dynkin diagram, and let (a0 , . . . , an ) be the numerical labels of S(A) in Table Aff from [Ka, pp. 48 – 49]. We denote by (a0∨ , . . . , an∨ ) the labels of the Dynkin diagram S(At ) of the dual algebra which is obtained from S(A) by reversing the direction of all arrows and keeping the same ˚ R, ˚ R˚ ∨ ) enumeration of the vertices. Let (h, R, R ∨ ) be a realization of A, and let (h, be the associated finite root system (which is a realization of the Cartan matrix A˚ = (ai j )1≤i, j≤n ). From these data one can construct an affine Kac-Moody algebra g, respectively, a finite Lie algebra g˚ , such that h, h˚ become the corresponding Cartan subalgebras and R, R˚ become the corresponding root systems. Note also that g˚ is a subalgebra of g. We refer to [Ka] for the details of this construction. If we denote ˚ we have the following by {αi }0≤i≤n a basis of R such that {αi }1≤i≤n is a basis of R, description: h∗ = h˚ ∗ + Rδ + R30 , Pn where δ = i=0 ai αi . The vector space h∗ has a canonical scalar product defined as follows: (αi , α j ) := di−1 ai j ,

(30 , αi ) := δi,0 a0−1 ,

and

(30 , 30 ) := 0,

with di := ai ai∨−1 and δi,0 as Kronecker’s delta. As usual, {αi∨ := di αi }0≤i≤n , {λi }1≤i≤n , and {λi∨ }1≤i≤n are the coroots, fundamental weights, and fundamental

302

BOGDAN ION

Ln Ln coweights, respectively. Denote by P = i=1 Zλi and Q˚ = i=1 Zαi the weight lattice, respectively, the root lattice, and let n 1 X ∨ X ∨ ρ := α = λi . 2 α∈ R˚ +

i=1

Given α ∈ R, x ∈ h∗ , let sα (x) := x −

2(x, α) α. (α, α)

The affine Weyl group W is generated by all sα (the simple reflections si = sαi are enough). The finite Weyl group W˚ is the subgroup generated by s1 , . . . , sn . An important role is played by θ = δ − a0 α0 . Remark that a0 = 1 in all cases except for (2) A = A2n , when a0 = 2. For s a real number, h∗s = {x ∈ h ; (x, δ) = s} is the level s of h∗ . We have h∗s = h∗0 + s30 = h˚ ∗ + Rδ + s30 . The action of W preserves each of the h∗s , and we can identify each of the h∗s canonically with h∗0 and obtain an (affine) action of W on h∗0 . If si ∈ W is a simple reflection, write si (·) for the regular action of si on h∗0 and si h·i for the affine action of si on h∗0 corresponding to the level-one action. These actions differ only for s0 : s0 (x) = sθ (x) + (x, θ)δ, s0 hxi = sθ (x) + a0−1 θ + (x, θ )δ − a0−1 δ. By si · we denote the affine action of W on h˚ ∗ , s0 · x = sθ (x) + a0−1 θ. We are interested in the cases when α0 is a short root. This happens precisely when the affine root system is twisted or simply laced untwisted. Under these conditions, we define the fundamental alcove as  A := x ∈ h˚ ∗ (x + 30 , αi∨ ) ≥ 0, 0 ≤ i ≤ n . The nonzero elements of O = P ∩ A are the so-called minuscule weights. Let us remark that each orbit of the affine action of W on P contains a unique λi ∈ A . (To keep the notation consistent, we set λ0 = 0.) In all that follows we assume our affine root system to be such that the affine simple root α0 is short. (This condition includes of course the case when all roots have the same length.)

NONSYMMETRIC MACDONALD POLYNOMIALS

303

1.2. The Bruhat order Let us first establish some notation. For each w in W , let l(w) be the length of a reduced (i.e., shortest) decomposition of w in terms of the si . We have l(w) = |5(w)|, where 5(w) = {α ∈ R+ | w(α) ∈ R− }. If w = s j p · · · s j1 is a reduced decomposition, then 5(w) = {α (i) | 1 ≤ i ≤ p} with α (i) = s j1 · · · s ji−1 (α ji ). For each weight λ, define λ− , respectively λ˜ , to be the unique element in W˚ λ, respectively W · λ, which is an antidominant weight, respec˚, tively, an element of O (that is a minuscule weight or zero), and define w˚ −1 λ ∈ W −1 wλ ∈ W to be the unique minimal length elements by which this is achieved. Also, for each weight λ, define λ+ to be the unique element in W˚ λ which is dominant, and denote by w◦ the maximal length element in W˚ . LEMMA 1.1 With the notation above, we have (i) 5(w˚ −1 λ ) = {α ∈ R˚ + | (λ, α) > 0}, (ii) 5(wλ−1 ) = {α ∈ R+ | (λ + 30 , α) < 0}.

Proof The proof is straightforward. See [C1, Theorem 1.4] for a full argument. The Bruhat order is a partial order on any Coxeter group. For its basic properties, see [H, Chapter 5]. Let us list a few of them. (The first two properties completely characterize the Bruhat order.) (1) For each α ∈ R+ we have sα w < w if and only if α is in 5(w−1 ) . (2) We have w0 < w if and only if w0 can be obtained by omitting some factors in a fixed reduced decomposition of w. (3) If w0 ≤ w, then either si w0 ≤ w or si w0 ≤ si w (or both). We can use the Bruhat order on W do define a partial order on the weight lattice: if λ, µ ∈ P, then by definition λ < µ if and only if wλ < wµ . 1.2 Let λ be a weight such that si · λ 6= λ for some 0 ≤ i ≤ n. Then wsi ·λ = si wλ . LEMMA

Proof Because l(si wλ ) = l(wλ ) ± 1 and l(si wsi ·λ ) = l(wsi ·λ ) ± 1, we have four possible situations depending on the choice of the signs in the above relations. The choice of a plus sign in both relations translates in αi 6∈ 5(wλ−1 ) and αi 6∈ 5(ws−1 ), which i ·λ by Lemma 1.1 and our hypothesis implies that (αi , λ + 30 ) > 0 and (αi , si · λ +

304

BOGDAN ION

30 ) > 0 (contradiction). The same argument shows that the choice of a minus sign in both relations is impossible. Now, we can suppose that l(si wλ ) = l(wλ ) + 1 and l(si wsi ·λ ) = l(wsi ·λ ) − 1, the other case being treated similarly. Using the minimal length properties of wλ and wsi ·λ , we can write l(wλ ) + 1 = l(si wλ ) ≥ l(wsi ·λ ) = l(si wsi ·λ ) + 1 ≥ l(wλ ) + 1, which shows that l(si wλ ) = l(wsi ·λ ). Our conclusion follows from the uniqueness of wsi ·λ . An immediate consequence is the following. 1.3 Let λ be a weight such that si · λ 6= λ for some 0 ≤ i ≤ n. Then si · λ > λ if and only if (αi , λ + 30 ) > 0. LEMMA

LEMMA 1.4 Let λ be a weight such that si · λ 6= λ for some 0 ≤ i ≤ n. Then w˚ si ·λ = si w˚ λ if i 6= 0 and w˚ s0 ·λ = sθ w˚ λ .

Proof We can prove the statement for i 6= 0 with the same arguments as in Lemma 1.2. The remaining statement was essentially proved in [S3, Lemma 3.3]. Definition 1.5 ˚ we say that the weight ν is a convex If λ and µ are weights such that λ − µ ∈ Q, ˚ combination of λ and µ if ν = (1 − τ )λ + τ µ such that 0 ≤ τ ≤ 1 and λ − ν ∈ Q. The following result was proved in [S4, Lemma 5.5] for a particular affine Weyl group, but the proof provided there works in general. LEMMA 1.6 Let λ be a weight such that si · λ > λ for some 0 ≤ i ≤ n. If ν is a proper convex combination of λ and si · λ, then ν < λ < si · λ.

For each weight λ, define λ = λ + w˚ λ (ρ). As a consequence of Lemma 1.4, we have the following.

NONSYMMETRIC MACDONALD POLYNOMIALS

305

PROPOSITION 1.7 Let λ be a weight such that si · λ 6= λ. Then

si · λ = si · λ. 1.3. Demazure module characters Recall that g is the Kac-Moody affine Lie algebra associated with the irreducible affine Cartan matrix A. For the results in this section, we refer to [Ku]. Let 3 be an integral dominant weight of g, and let V = V (3) be the unique irreducible highest weight g-module with highest weight 3. For each w ∈ W the weight space Vw(3) is one-dimensional. Consider E w (3), the b-module generated by Vw(3) , where b is the Borel subalgebra of g. The E w (3), called the Demazure modules, are finitedimensional vector spaces. If λ is an element of O , then λ + 30 is a dominant weight. In such a case we write E w (λ) for E w (λ + 30 ). To a Demazure module E w (3) we can associate its character X

χ E w (3) = dim E w (3)ϒ · eϒ , ϒ weight

which can be regarded as an element in P := C[q, q −1 ][eµ ; µ ∈ P] after we ignore the e30 factor and after we set q = e−δ . Definition 1.8 For each 0 ≤ i ≤ n, define an operator 1i acting on P , 1i e3 =

e3 − e−αi esi h3i . 1 − e−αi

Let w = si1 · · · si j be a reduced decomposition. Then we can define the operator 1w := 1i1 · · · 1i j . (The definition of 1w does not depend on the choice of the reduced decomposition.) 1.9 Let λ be an element of O . Then THEOREM


χ E w (λ) = 1w (eλ ). Theorem 1.9 is a special case of the Demazure character formula for Kac-Moody algebras, which has been proved in full generality by S. Kumar and independently by O. Mathieu. We refer to [Ku, Theorem 3.4] for the proof. The construction of the Demazure modules E˚ w˚ (λ) for the Lie algebra g˚ is completely analogous. (The role of b is played here by b˚ the Borel subalgebra of g˚ .)

306

BOGDAN ION

2. Nonsymmetric Macdonald polynomials In what follows we consider our root system to be reduced. Recall that in this case a0 = 1 and θ is the highest short root of the associated finite root system. The case of a nonreduced root system is treated in Section 3. 2.1. The double affine Hecke algebra We introduce a field F (of parameters) as follows: fix indeterminates q and t0 , . . . , tn such that ti = t j if and only if di = d j , let m be the lowest common denominator of the rational numbers {(αi , λ j ) | 1 ≤ i, j ≤ n}, and let F denote the field of rational 1/2 functions in q 1/m and ti . Because in our case there are at most two different root lengths, we also use the notation tl , ts for ti if the corresponding simple root is long, respectively, short. The algebra R = F[eλ ; λ ∈ P] is the group F-algebra of the lattice P, and S is the subalgebra of R consisting of elements invariant under the finite Weyl group. For further use we also introduce the following group of F-algebras of the root ˚ and R X := F[X β ; β ∈ Q]. ˚ SY is the subalgebra of RY lattice: RY := F[Yµ ; µ ∈ Q] consisting of elements invariant under the finite Weyl group. Definition 2.1 The affine Hecke algebra H is the F-algebra generated by elements T0 , . . . , Tn with the following relations. (i) The Ti satisfy the same braid relations as the si . (ii) For 0 ≤ i ≤ n we have 1/2

Ti2 = (ti

−1/2

− ti

)Ti + 1.

The elements T1 , . . . , Tn generate the finite Hecke algebra H˚ . There are natural bases of H and H˚ : {Tw }w indexed by w in W and in W˚ , respectively, where Tw = Til · · · Ti1 if w = sil · · · si1 is a reduced expression of w in terms of simple reflections. There is another important description of the affine Hecke algebra due to G. Lusztig [L2]. PROPOSITION 2.2 The affine Hecke algebra H is generated by the finite Hecke algebra and the group algebra RY such that the following relation is satisfied for any µ in the root lattice and any 1 ≤ i ≤ n: 1/2

Yµ Ti − Ti Ysi (µ) = (ti Remark 2.3 In this description, T0−1 = Yθ Tsθ .

−1/2

− ti

)

Yµ − Ysi (µ) . 1 − Yαi

NONSYMMETRIC MACDONALD POLYNOMIALS

307

Following Macdonald [M], we call the family of commuting operators RY ⊂ H Cherednik operators. In order to state the next result, we need the following notation: for µ, β ∈ Q˚ and k ∈ Z, X β+kδ := q −k X β and Yµ+kδ := q k Yµ . For the next results we refer to Cherednik [C1], [C3]. Definition 2.4 The double affine Hecke algebra H d is the F-algebra generated by the affine Hecke algebra H and the group algebra R X such that the following relation is satisfied for any β in the root lattice and any 0 ≤ i ≤ n: 1/2

−1/2

Ti X β − X si (β) Ti = (ti

− ti

)

X β − X si (β) . 1 − X −αi

The following formulas define a faithful representation of H d on R : π(Ti )eλ = ti

1/2 si (λ)

e

π(X β )eλ = eλ+β ,

1/2

+ (ti

−1/2

− ti

)

eλ − esi (λ) , 1 − e−αi

0 ≤ i ≤ n,

˚ β ∈ Q.

THEOREM 2.5 Define Th0i = T0−1 X α0 . Then for all µ ∈ Q˚ and all λ ∈ P, 1/2

Yµ Th0i − Th0i Ys0 (µ) = (t0

−1/2

− t0

)

Yµ − Ys0 (µ) , 1 − Yα0

π(Th0i )eλ = t0 es0 hλi + (t0 1/2

1/2

−1/2

− t0

)

eλ − es0 hλi . 1 − e−α0 (1)

The irreducible affine root systems for which the affine simple root is long are Bn , (1) (1) (1) Cn , F4 , and G 2 . For these root systems the formula of the element of the double affine Hecke algebra which plays the same role as Th0i takes a different form (see [I]), which makes the computation of its action on R more difficult. To avoid cumbersome notation, we set Thii = Ti for i 6= 0 and write H eλ in place of π(H )eλ for any H ∈ H d . The following result follows directly from Lemma 1.6. THEOREM 2.6 Suppose that λ ≤ γ ≤ si · γ for some weights λ, γ and 0 ≤ i ≤ n. Then

Thii eλ = ti

1/2 si hλi

e

+ lower terms,

where by lower terms we mean a combination of eβ with β < si · λ.

308

BOGDAN ION

2.2. Macdonald polynomials Cherednik defined a certain scalar product on R (see [C1] for details) for which all operators in H became unitary operators. In particular, the adjoint of Yµ is Y−µ . By q(µ+kδ,λ) we denote the element of F, q

k+(µ,λ)

n Y

−(µ,w˚ λ (λi∨ ))

ti

.

i=1

For each λ ∈ P we can construct an F-algebra morphism ev(λ) : RY → F which sends Yµ to q(µ,λ) . If f is an element of RY , we write f (λ) for ev(λ)( f ). Macdonald defined a basis {Pλ (q, t)} of S which is indexed by antidominant weights and which is completely characterized by the equations f · Pλ = f (λ)Pλ

(1)

for any f ∈ SY and by the condition that the coefficient of eλ in Pλ (q, t) is 1. The elements of this basis are called symmetric Macdonald polynomials. Recently, a nonsymmetric version of the Macdonald polynomials was introduced by Opdam [O] in the differential case, by Macdonald [M] for ti = q k , k ∈ Z+ , and by Cherednik [C2] in the general (reduced) case, and some of their properties were studied. For each weight λ there is a unique element E λ (q, t) ∈ R satisfying the following conditions: E λ = eλ + lower terms,

(2)

µ

(3)

(E λ , e ) = 0

for all µ < λ.

They form an F-basis of R , and they are the common eigenfunctions of the Cherednik operators. In what follows we find an explicit recursion formula for the nonsymmetric Macdonald polynomials. In the course of doing that, we give a more transparent proof of their existence and uniqueness. For all 0 ≤ i ≤ n, let us introduce the following elements of H d , called intertwiners: 1/2 1/2 Ii := Thii (1 − Yαi ) − (ti − ti ). The intertwiners were first introduced by F. Knop and Sahi (see [Kn], [KS], [S1]) for GLn and then by Cherednik [C3] in the general (reduced) case. Their importance is the following: for any µ in the root lattice, we have Yµ Ii = Ii Ysi (µ) .

(4)

This easily follows from Proposition 2.2 and Theorem 2.5. The next results can be proved following closely the ideas in [S2], where the nonreduced case was considered. For every weight λ, define  Rλ = f ∈ R Yµ f = q(µ,λ) f for any µ ∈ Q˚ .

NONSYMMETRIC MACDONALD POLYNOMIALS

309

THEOREM 2.7 Let λ be a weight such that si · λ 6= λ. Then Ii : Rλ → Rsi ·λ is a linear isomorphism.

Proof Let f be any element of Rλ . Using the intertwining relation (4) and Proposition 1.7, we get Yµ (Ii f ) = q(µ,si ·λ) Ii f. Therefore Ii f is an element of Rsi ·λ . A short computation shows that Ii2 = ti + ti−1 − (Yαi + Y−αi ); therefore Ii2 acts as a constant on Rλ . It is easy to see that our hypothesis implies that this constant is nonzero, showing that Ii2 , and consequently Ii , is an isomorphism. 2.8 The spaces Rλ are one-dimensional. THEOREM

Proof The proof is very similar to the proof of the corresponding result in [S2, Theorem 6.1]. The only difference is that we have to use the fact that O is a set of representatives for the orbits of the affine action of W on P and that eλ is in Rλ for λ ∈ O . From the proof also follows that an element in Rλ is uniquely determined by the coefficient of eλ in f . This result makes possible the following definition. Definition 2.9 For any weight λ, define the nonsymmetric Macdonald polynomial E λ (q, t) to be the unique element in Rλ in which the coefficient of eλ is 1. If k ∈ Z, then denote E λ+kδ (q, t) = q −k E λ (q, t). For each antidominant weight λ, we write R λ for the subspace of R spanned by {E µ | µ ∈ W˚ λ}. The connection with the symmetric Macdonald polynomials is the following. COROLLARY 2.10 The polynomial Pλ (q, t) can be characterized as the unique W˚ -invariant element in R λ for which the coefficient of eλ equals 1.

Proof The result follows from the characterization (1).

310

BOGDAN ION

Definition 2.11 Let C be the element of the finite Hecke algebra defined by X
−1 X C := χ(Tw )2 χ(Tw )Tw , w∈W˚

w∈W˚ 1/2

where χ is the one-dimensional representation of H˚ defined by χ (Ti ) = ti

.

COROLLARY 2.12 The operator π(C) is a projection from R λ to FPλ .

Proof 1/2 An easy calculation, as in [S1, Lemma 2.5], shows that Ti C = ti C for any 1 ≤ i ≤ 1/2 n; hence Ti (C f ) = ti C f for all f ∈ R . This implies that C f is W˚ -invariant, and so it must be a multiple of Pλ . Moreover, C acts as the identity on S . For any weight λ and any 0 ≤ i ≤ n, define the operator G i,λ (q, t) as follows: −1/2

G i,λ := ti

Thii

if (λ + 30 , αi ) = 0

and G i,λ := (1 − q−(αi ,λ) )ti

−1/2

Thii + q−(αi ,λ) (1 − ti−1 ) if (λ + 30 , αi ) 6= 0.

THEOREM 2.13 Let λ be a weight such that (λ + 30 , αi ) > 0. Then

G i,λ E λ = (1 − q−(αi ,λ) )E si hλi .

(5)

If (λ + 30 , αi ) = 0, then G i,λ E λ = E λ . Proof When (λ + 30 , αi ) = 0, the statement follows straightforwardly from (2), (3), and Theorem 2.6. For the remaining case, using Theorem 2.7, all we need is to compute the coefficient of esi hλi in G i,λ E λ , which by Theorem 2.6 can be shown to be (1 − q−(αi ,λ) ). 2.3. The specialization at t = ∞ Our goal is to define the specialization of the polynomials E λ (q, t) at t = ∞ (which means t −1 = 0) and to obtain recursion formulas for them as in Theorem 2.13. In order to do this, we have to closely examine the coefficients of the E λ and make

NONSYMMETRIC MACDONALD POLYNOMIALS

311

sure that their limit exists. In fact, we can suitably renormalize the E λ so that all the coefficients in this renormalization are polynomials in ti−1 and the normalizing factor approaches 1 when t tends to infinity. This shows that the limit of each of the coefficients of the E λ exists and that it is bounded. Recall that wλ is the unique minimal length element of W such that wλ · λ˜ = λ. Let wλ = s jl · · · s j1 be a reduced decomposition. Then  5(wλ ) = α (i) := s j1 · · · s ji−1 (α ji ) 1 ≤ i ≤ l . (6) This means in particular that α ( j) ∈ R+ and wλ (α ( j) ) ∈ R− . Define λ(i) := s ji−1 · · · s j1 · λ˜

(7)

for any 1 ≤ i ≤ l + 1. Therefore λ(1) = λ˜ and λ(l+1) = λ. The key property of the λ(i) is (λ(i) + 30 , α ji ) > 0. (8) This easily follows from (6). Moreover, (8) implies that α ji ∈ 5(w˚−1 λ(i)) if ji 6= 0, −1 −1 ˚ meaning that w˚ λ(i)(α ji ) is in R− , respectively, that θ 6∈ 5(w˚ λ(i)) if ji = 0, meaning that w˚−1 λ(i)(θ) is in R˚ + .

Now, for all 1 ≤ j ≤ l, all the exponents in the monomial q(α ji ,λ(i) ) are positive integers and at least one of exponents the ti is nonzero. Define the renormalization of E λ (q, t) to be l Y (1 − q−(α ji ,λ(i) ) )E λ (q, t). i=1

This formula (modulo a q factor) is obtained by applying the recursion formula (5) ˜ successively, starting with eλ . From this description it is clear that the powers of the ti appearing in the expansion of this renormalization of E λ (q, t) are all negative, and therefore our desired specialization at t = ∞ is well defined. We denote by E λ (q, ∞) this specialization. This renormalization does not depend on the choice of the reduced decomposition of wλ . Remark also that the coefficient of eλ in E λ (q, ∞) is 1. For each antidominant weight λ, we write R λ (∞) for the linear subspace spanned by {E µ (q, ∞) | µ ∈ W˚ λ}. The polynomial Pλ (q, ∞) is defined to be the unique W˚ invariant element in R λ (∞) for which the coefficient of eλ equals 1. 3. Nonsymmetric Koornwinder polynomials In this section we consider the case of a nonreduced root system. Recall that in this case A = A2(2n) , a0 = 2, θ is the highest root, and O = {0}.

312

BOGDAN ION

3.1. The recursion relation The results in this section are due to Sahi [S2], [S4]. We introduce the field F as follows: fix indeterminates q, u = (u 0 , u n ) and t0 , . . . , tn identified as before; the field F is the field of rational functions in their square roots. We also define 1/2 1/2

a = tn u n , c=

1/2 1/2 q 1/2 t0 u 0 ,

1/2 −1/2

b = −tn u n d=

,

1/2 −1/2 −q 1/2 t0 u 0 .

Note that in this case we have three different root lengths; therefore t = (ts , tm , tl ), where ts = t0 , tl = tn , and tm = ti for any i 6= 0, n. As before, R = F[eλ ; λ ∈ P] is the group F-algebra of the lattice P and S is the subalgebra of R consisting of elements invariant under the finite Weyl group. Also, define RY := F[Yµ ; µ ∈ P] and R X := F[X β ; β ∈ P]. SY is the subalgebra of RY consisting of elements invariant under the finite Weyl group. The lattice P can be identified with Zn such that the scalar product we defined in Section 1.1 is the canonical scalar product on Rn . If ε1 , . . . , εn are the unit vectors in Zn , then our choice of the basis for the affine root system is 1 α0 = δ + ε1 , αi = −εi + εi+1 , αn = −2εn . 2 The double affine Hecke algebra in this case has a more complicated description (see [S2] for details). We describe here only its action on R : −1/2 (1 − ce

T0 eλ := t0 eλ + t0 1/2

−ε1 )(1 − de−ε1 )

1 − qe−2ε1

(es0 (λ) − eλ ),

Th0i eλ := T0−1 eλ+α0 , Thii eλ = Ti eλ := ti

1/2 λ

−αi ) −1/2 (1 − ti e (esi (λ) −α (1 − e i )

e + ti

−1/2 (1 − ae

Thni eλ = Tn eλ := tn eλ + tn 1/2

− eλ ),

εn )(1 − beεn )

1 − qe2εn

i 6= 0, n,

(esn (λ) − eλ ).

The commutative algebra RY embeds in the Hecke algebra as follows: −1 Yεi = (Ti · · · Tn−1 )(Tn · · · T0 )(T1−1 · · · Ti−1 ).

The action of RY can be simultaneously diagonalized, and the nonsymmetric Koornwinder polynomials E λ (q, t, u) are the corresponding eigenbasis. The eigenvalues are given as follows: by q(µ+kδ,λ) we denote the element of F, q k+(µ,λ) (t0 tn )−(µ,w˚ λ (λn )) ∨

n−1 Y

−(µ,w˚ λ (λi∨ ))

ti

.

i=1

For each λ ∈ P we can construct an F-algebra morphism ev(λ) : RY → F which sends Yµ to q(µ,λ) . If f is an element of RY , we write f (λ) for ev(λ)( f ).

NONSYMMETRIC MACDONALD POLYNOMIALS

313

The symmetric Koornwinder polynomials {Pλ (q, t, u)} form a basis of S which is indexed by antidominant weights. They are completely characterized by the equation f · Pλ = f (λ)Pλ

(9)

for any f ∈ SY and the condition that the coefficient of eλ in Pλ (q, t, u) be equal to 1. In the same manner as in Section 2.2, we define for any weight λ the vector spaces Rλ and R λ . PROPOSITION 3.1 The polynomial Pλ (q, t, u) can be characterized as the unique W˚ -invariant element in R λ for which the coefficient of eλ equals 1.

For any weight λ and any 0 ≤ i ≤ n such that (λ + 30 , αi ) = 0, define the operator G i,λ (q, t) as follows: −1/2 G i,λ := ti Thii . If (λ + 30 , αi ) 6= 0, we define G i,λ := (1 − q−(αi ,λ) )ti

−1/2

Ti + q−(αi ,λ) (1 − ti−1 ) for i 6= 0, n,

−1/2

(1 − q−(αn ,λ) )Thni + q−(αn ,λ) (tn 1/2 −1/2
+ q−(1/2)(αn ,λ) (u n − u n ) ,

−1/2

(1 − q−2(α0 ,λ) )Th0i + q−2(α0 ,λ) (u 0 1/2 −1/2
+ q−(α0 ,λ) (u n − u n ) .

G n,λ := tn

G 0,λ := t0

1/2

−1/2

− tn

1/2

)

−1/2

− u0

)

THEOREM 3.2 Let λ be a weight such that (λ + 30 , αi ) > 0. Then

G i,λ E λ = (1 − q−(αi ,λ)−δi,0 (α0 ,λ) )E si hλi .

(10)

If (λ + 30 , αi ) = 0, then G i,λ E λ = E λ . 3.2. The specialization at u = (t0 , 1), t = ∞ First, there is of course no problem in specializing u 0 := t0 and u n = 1. The problem arises, as in Section 2.3, when we want to specialize t = ∞. One can follow closely the argument in Section 2.3 to examine the coefficients of the E λ . We just state the corresponding result in this case. Recall that wλ is the unique minimal length element of W such that wλ · 0 = λ, {α (i) }, and {λ(i) } are elements defined as in equations (6) and (7).

314

BOGDAN ION

Define the renormalization of E λ (q, t, u) to be l Y (1 − q−(α ji ,λ(i) )−δ ji ,0 (α0 ,λ(i) ) )E λ (q, t, u). i=1

This formula (modulo a q factor) is obtained by applying the recursion formula (10) successively, starting with 1. The powers of the ti appearing in the expansion of this renormalization after the substitution u = (t0 , 1) are all negative, and the normalizing factor tends to 1 when t approaches infinity. Therefore our desired specialization at t = ∞ is well defined. We denote by E λ (q, ∞) this specialization. Note that the coefficient of eλ in E λ (q, ∞) equals 1. For each antidominant weight λ, we write R λ (∞) for the linear subspace spanned by {E µ (q, ∞) | µ ∈ W˚ λ}. The polynomial Pλ (q, ∞) is defined to be the unique W˚ -invariant element in R λ (∞) for which the coefficient of eλ equals 1. 4. The representation-theoretical interpretation In this section we make no more reference to the fact that the root system in question is reduced or not, but, depending on the case, we use the notation E λ (q, ∞) to refer to the specialized versions of the nonsymmetric Macdonald polynomials or nonsymmetric Koornwinder polynomials. 4.1. Proof of Theorem 1 The strategy is to study the degeneration of recursion formulas (5) and (10) for the polynomials E λ (q, ∞) and then to relate them to the Demazure character formula (Theorem 1.9). The crucial remark is that we are interested only in the action of the operators G i,λ (q, t) on the renormalization of the E λ (q, t) when (λ+30 , αi ) ≥ 0. We see, after an examination of the operator G i,λ (q, t) in this situation, that the powers of ti appearing in the description of its action are negative or zero. Because the same is true for the renormalization of E λ (q, t), we can first make the specialization at t = ∞. Moreover, the operators G i,λ (q, ∞) do not depend on λ anymore. In fact, G i,λ (q, ∞) coincide with the Demazure operators 1i . We are ready to state the following. 4.1 Let λ be a weight such that (λ + 30 , αi ) ≥ 0. Then THEOREM

1i E λ (q, ∞) = q −(30 ,si hλi) E si ·λ (q, ∞).

(11)

Proof The statement is obvious for (λ + 30 , αi ) = 0. Now, we know from Lemma 1.3 that if (λ + 30 , αi ) > 0, we have l(wsi ·λ ) = l(si wλ ) = l(wλ ) + 1.

NONSYMMETRIC MACDONALD POLYNOMIALS

315

Therefore if wλ = s j p · · · s j1 is a reduced decomposition, wsi ·λ = si s j p · · · s j1 is also reduced. Hence, using the definitions of E λ (q, ∞) and E si ·λ (q, ∞) and recursion formulas (5) and (10), our conclusion follows. An immediate consequence of Theorem 4.1 is that 1wλ eλ = q −(30 ,wλ hλi) E λ (q, ∞). ˜

˜

Theorem 1 follows by comparing this formula with Theorem 1.9. A simple consequence of Theorem 1 is that if we expand E λ (q, ∞) in terms of monomials, the coefficients that appear are polynomials in q −1 with positive integer coefficients. 4.2. Proof of Theorem 2 Let us begin with a characterization of Pλ (q, ∞). If λ is antidominant, then (λ, αi ) ≤ 0 and Theorem 4.1, together with 1i2 = 1i , shows that 1i E λ (q, ∞) = E λ (q, ∞). This immediately implies that E λ (q, ∞) is W˚ -invariant. 4.2 If λ is an antidominant weight, then THEOREM

Pλ (q, ∞) = E λ (q, ∞). ˜ Now, because Pλ (q, ∞) is essentially the character of the Demazure module E wλ (λ), ˚ ˜ the W -invariance of Pλ (q, ∞) translates into saying that E wλ (λ) decomposes into a direct sum of simple g˚ -modules. Let us write M E wλ (λ˜ ) = E wλ (λ˜ ) j , j≥0

where E wλ (λ˜ ) j is the direct sum of weight spaces whose weights are of the form µ + jδ + (30 , wλ hλ˜ i)δ with integer j and µ ∈ P. Since δ is W˚ -invariant, each of the E wλ (λ˜ ) j decomposes as a direct sum of simple g˚ -modules. If χµ is the character of Vµ , the irreducible g˚ -module with highest weight µ, then X j
˜ χ E wλ (λ˜ ) j = q −(30 ,wλ hλi)− j cλµ χµ . µ

j Here cλµ is the multiplicity of Vµ in E wλ (λ˜ ) j . Summing up, we find the polynomials in q −1 with positive integer coefficients such that X Pλ (q, ∞) = dλµ (q)χµ . µ≤λ

316

BOGDAN ION

The restriction on the sum comes from the triangular properties of Pλ . Let us remark that the positive integers dλµ (1) are the multiplicities of the irreducible g˚ -modules in ˜ Also, dλλ (q) = 1. the Demazure module E wλ (λ). 4.3. Proof of Theorem 3 On one hand, because the coefficients of the expansion of E λ (q, ∞) in terms of monomials are polynomials in q −1 with positive integer coefficients, their limit at q → ∞ exists. We denote E λ (∞, ∞) = lim E λ (q, ∞). q→∞

On the other hand, using Theorem 1, we can see that
E λ (∞, ∞) = χ E wλ (λ˜ )0 , where E wλ (λ˜ )0 is the direct sum of weight spaces whose weights are of the form ˚ µ + (30 , wλ hλ˜ i)δ with µ ∈ P. It can be easily seen that E wλ (λ˜ )0 is a b-module. ˚ ˚ ˜ Our conclusion follows if we prove that E wλ (λ)0 is isomorphic to E w˚ λ w◦ (λ+ ) as bmodules. As explained in the proof of the Theorem 2, the vector space E wλ− (λ˜ )0 is also a g˚ -module. 4.3 The g˚ -module E wλ− (λ˜ )0 is the irreducible representation of g˚ with highest weight λ+ . ˚ ˜ 0 and E˚ w˚ w (λ+ ) are isomorphic. Furthermore, the b-modules E w (λ) THEOREM

λ

λ

◦

Proof By Theorem 2, we know that the irreducible representation of g˚ with highest weight λ+ occurs in the decomposition of E wλ− (λ˜ )0 with multiplicity one. Let us denote by V˚ the copy of the irreducible representation of g˚ with highest weight λ+ embedded in E wλ− (λ˜ )0 and by V the irreducible representation of g with highest weight 3 = ˚ λ˜ + 30 . It is easy to see that E wλ− (λ˜ )0 is the b-module generated by the weight space −1 Vw(3) , where w = w˚ λ wλ . From the fact that the space Vw(3) is one-dimensional and from w(3) = λ− + (30 , wλ hλ˜ i), we deduce that Vw(3) is the lowest weight space of V˚ , and therefore V˚ = E wλ− (λ˜ )0 , ˚ both being equal to the b-module generated by the weight space Vw(3) . By the same ˚ argument, the b-module E wλ (λ˜ )0 is generated by the one-dimensional weight space Vwλ (3) = V˚ w˚ λ w◦ (λ+ ) ,

NONSYMMETRIC MACDONALD POLYNOMIALS

317

˚ which also generates E˚ w˚ λ w◦ (λ+ ) as a b-module. Our conclusion follows. The proof of Theorem 3 is now complete. Acknowledgment. I want to acknowledge my deep gratitude to Siddhartha Sahi for his generous and inspiring guidance. References [C1] [C2] [C3] [H] [I] [Ka] [Kn] [KS] [Ku] [L1] [L2] [M]

[Ma] [O] [S1] [S2]

I. CHEREDNIK, Double affine Hecke algebras and Macdonald’s conjectures, Ann. of

Math. (2) 141 (1995), 191 – 216. MR 96m:33010 300, 303, 307, 308 , Nonsymmetric Macdonald polynomials, Internat. Math. Res. Notices 1995, 483 – 515. MR 97f:33032 299, 300, 308 , Intertwining operators of double affine Hecke algebras, Selecta Math. (N.S.) 3 (1997), 459 – 495. MR 99g:33036 300, 307, 308 J. E. HUMPHREYS, Reflection Groups and Coxeter Groups, Cambridge Stud. Adv. Math. 29, Cambridge Univ. Press, Cambridge, 1990. MR 92h:20002 303 B. ION, Involutions of double affine Hecke algebras, to appear in Compositio Math., preprint, arXiv:math.QA/0111010 300, 307 V. G. KAC, Infinite-dimensional Lie algebras, 3d ed., Cambridge Univ. Press, Cambridge, 1990. MR 92k:17038 301 F. KNOP, Integrality of two variable Kostka functions, J. Reine Angew. Math. 482 (1997), 177 – 189. MR 99j:05189c 308 F. KNOP and S. SAHI, A recursion and a combinatorial formula for Jack polynomials, Invent. Math. 128 (1997), 9 – 22. MR 98k:33040 308 S. KUMAR, Demazure character formula in arbitrary Kac-Moody setting, Invent. Math. 89 (1987), 395 – 423. MR 88i:17018 305 G. LUSZTIG, Green polynomials and singularities of unipotent classes, Adv. Math. 42 (1981), 169 – 178. MR 83c:20059 , Affine Hecke algebras and their graded version, J. Amer. Math. Soc. 2 (1989), 599 – 635. MR 90e:16049 306 I. G. MACDONALD, Affine Hecke algebras and orthogonal polynomials, Ast´erisque 237 (1996), 189 – 207, S´eminaire Bourbaki 1994/95, exp. no. 797. MR 99f:33024 307, 308 O. MATHIEU, Formules de caract`eres pour les alg`ebres de Kac-Moody g´en´erales, Ast´erisque 159 – 160, Soc. Math. France, Montrouge, 1988. MR 90d:17024 E. M. OPDAM, Harmonic analysis for certain representations of graded Hecke algebras, Acta Math. 175 (1995), 75 – 121. MR 98f:33025 299, 308 S. SAHI, Interpolation, integrality, and a generalization of Macdonald’s polynomials, Internat. Math. Res. Notices 1996, 457 – 471. MR 99j:05189b 308, 310 , Nonsymmetric Koornwinder polynomials and duality, Ann. of Math. (2) 150 (1999), 267 – 282. MR 2002b:33018 300, 308, 309, 312

318

[S3] [S4]

[Sa]

BOGDAN ION

, A new formula for weight multiplicities and characters, Duke Math. J. 101 (2000), 77 – 84. MR 2000j:17009 304 , “Some properties of Koornwinder polynomials” in q-Series from a Contemporary Perspective (South Hadley, Mass., 1998), Contemp. Math. 254, Amer. Math. Soc., Providence, 2000, 395 – 411. MR 2001h:33025 300, 304, 312 Y. B. SANDERSON, On the connection between Macdonald polynomials and Demazure characters, J. Algebraic Combin. 11 (2000), 269 – 275. MR 2001h:17018 300, 301

Department of Mathematics, Princeton University, Princeton, New Jersey 08544, USA; [email protected]

ELLIPTIC GENERA OF SINGULAR VARIETIES LEV BORISOV and ANATOLY LIBGOBER

Abstract The notions of orbifold elliptic genus and elliptic genus of singular varieties are introduced, and the relation between them is studied. The elliptic genus of singular varieties is given in terms of a resolution of singularities and extends the elliptic genus of Calabi-Yau hypersurfaces in Fano Gorenstein toric varieties introduced earlier. The orbifold elliptic genus is given in terms of the fixed-point sets of the action. We P show that the generating function for the orbifold elliptic genus Ellorb (X n , 6n ) p n for symmetric groups 6n acting on n-fold products coincides with the one proposed by R. Dijkgraaf, G. Moore, E. Verlinde, and H. Verlinde. The two notions of elliptic genera are conjectured to coincide. 1. Introduction This work started as an attempt to understand the beautiful formula for the generating function for the orbifold elliptic genera of symmetric products due to Dijkgraaf, Moore, Verlinde, and Verlinde which follows (see [19]): X n≥0

p Ellorb (X /6n ; y, q) = n

n

∞ Y Y i=1 l,m

1 (1 −

pi y l q m )c(mi,l)

.

(1.1)

Here X is a K¨ahler manifold, 6n is the symmetric group acting on the n-fold product, P and c(m, l) are the coefficients of the elliptic genus m,l c(m, l)y l q m of X . The problem is that the orbifold elliptic genus is defined in physical terms, and the arguments given in [19] do not lend themselves to a translation into a mathematical proof. The two-variable elliptic genus is a very compelling invariant, for the discussion of which we refer to [12]. Here we just note that it is a holomorphic function on the product of C and the upper half-plane which is attached to an (almost) complex manifold and that it is a weak Jacobi form if the manifold is Calabi-Yau. For Calabi-Yau manifolds of a dimension smaller than 12 or equal to 13, the elliptic genus can be expressed in terms of the Hirzebruch χ y genus, but in general the former contains more information than the latter. In all dimensions the elliptic genus specializes into the Hirzebruch χ y genus, and in particular into the topological Euler characteristic, DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 2, Received 4 April 2001. Revision received 6 December 2001. 2000 Mathematics Subject Classification. Primary 14J32, 14J17, 32S45; Secondary 55N34. 319

320

BORISOV and LIBGOBER

the holomorphic Euler characteristic, the signature, and so on. Special cases of formula (1.1) for these invariants have been proved mathematically for some time. For example, it was shown in [28], using the Macdonald formula (see [31]), that if a finite group G acts on a manifold X and 1 X eorb (X, G) := e(X f,g ) (1.2) |G| f g=g f

(summation is over all pairs of commuting elements; X f,g is the set of fixed points of both f and g), then n=∞ X

eorb (X n , 6n ) =

Y i

n=0

1 . (1 − t i )e(X )

(1.3)

On the other hand, in [24] (see also [21]) it was found that the generating series for the χ y genera of Hilbert schemes of a surface X is given by n=∞ X n=0

χ−y (X

[n]

∞ X χ−y m (X ) p m  ) p = exp . (1 − (yp)m ) m n

(1.4)

m=1

It was observed in [28] that in the cases when a crepant resolution for X/G does exist, the McKay correspondence (see [33]) can be used to prove that the Euler characteristic of such a resolution coincides with the orbifold Euler characteristic. In [7] this idea was used in a more general case of the χ y genus, with an appropriately defined orbifold χ y genus. In the case when X is a surface, the Hilbert scheme provides such a resolution (see [22]), and hence the left-hand side of (1.4) coincides with the generating function for the orbifold χ y genus of symmetric products of X . Therefore, (1.4) can be viewed as a specialization of (1.1). This brings in the basic question: How are the orbifold Euler characteristic and the orbifold χ y genus (or more generally, the orbifold elliptic genus of an action on a variety) related to the corresponding invariants of an arbitrary, not necessarily crepant, resolution of the singularities of the orbifold? This question has been addressed in several papers (see, e.g., [5], [7], [18]). Paper [5] contains mathematical definitions of the orbifold E-function and of an E-function of singular varieties calculated via resolutions, which is called a stringy E-function. The E-function of a smooth manifold is equivalent to the data given by the Hodge numbers of the manifold, and it specializes to the χ y genus. The stringy E-function is defined for singular varieties with log-terminal singularities and more generally for log-terminal pairs. Works [5] and [18] show that the orbifold E-function for a pair (X, G) coincides with the stringy Efunction for the pair (X/G, image of ramification divisor). The published version of [5] has a gap in its canonical abelianization algorithm, but it has now been corrected by Batyrev [6].

ELLIPTIC GENERA OF SINGULAR VARIETIES

321

In this paper, two notions of the elliptic genus for singular varieties are proposed. The first is called the singular elliptic genus and is defined for pairs (variety, divisor). The singular elliptic genus specializes to the χ y genus derived from the stringy E-function of [5]. The second notion of elliptic genus, called the orbifold elliptic genus, is defined for any pair (X, G) of a manifold and a finite group of its automorphisms. The orbifold elliptic genus specializes to the χ y genus derived from the orbifold E-function. We conjecture that the two elliptic genera coincide for (X/G, image of ramification divisor) and (X, G), up to an explicit normalization factor. The advantage of the orbifold elliptic genus is that it is well suited for the mathematical proof of formula (1.1). On the other hand, the singular elliptic genus provides an interesting new invariant of singular varieties. Instead of the non-Archimedian integrals over spaces of arcs techniques of [5] and [18], we use the recent result in factorization of birational maps into a sequence of smooth blow-ups and blow-downs (see [1]). The content of the paper is as follows. In Section 2 we collect some standard definitions and results that are relevant to the subject but may not be familiar to the reader. In Section 3 we define the singular elliptic genus of a Q-Gorenstein complex projective variety Z as follows. If π : Y → Z is a resolution of singularities of Z and P αk ∈ Q are defined from the relation K Y = π ∗ K Z + αk E k , then Z Y (yl /(2πi))θ(yl /(2πi) − z)θ 0 (0)  c Y (Z ; z, τ ) := Ell θ(−z)θ(yl /(2πi)) Y l Y θ(e /(2πi) − (α + 1)z)θ(−z)  k k × , θ(ek /(2πi) − z)θ(−(αk + 1)z) k

where θ(z, τ ) is the Jacobi theta function and yl are Chern roots of Y and ek = c1 (E k ). c Y (Z ; z, τ ) depends only on Z (rather than on the desingularization It is shown that Ell Y ). Moreover, this definition is extended to pairs (variety, divisor), and the singular elliptic genus has the transformation properties of a Jacobi form if the pair satisfies a natural Calabi-Yau condition. Some difficulties arise only when some αk equal (−1) and we assume that the pairs are Kawamata log-terminal. One application of the singular elliptic genus is to the problem raised by M. Goreski and R. McPherson (see [9]). They were trying to determine which Chern numbers can be defined for singular spaces so that they are invariant under small resolutions. B. Totaro found a remarkable connection between this problem and the elliptic genus. In [35] he showed that such Chern numbers must be among the combinations of the coefficients of the two variable elliptic genus by showing that these are the only Chern numbers invariant under the classical flops. As a corollary of our definition of singular elliptic genus, we show that the elliptic genera of any two IH-small resolutions (or, more generally, two crepant resolutions) of a singular variety coincide, which in a sense completes

322

BORISOV and LIBGOBER

the paper of Totaro. Unfortunately, most varieties do not admit such resolutions, and it appears that Chern numbers may not be a good invariant to look for because singular elliptic genera generally do not lie in the span of the elliptic genera of smooth varieties. However, coefficients of Taylor expansions of elliptic genera do provide an analog of Chern numbers for singular varieties. In Section 4 we propose a definition of an orbifold elliptic genus which does not use the resolution of singularities but uses only information about the manifold and the fixed-point sets. Let G be a finite group acting on a manifold X . For h ∈ G, let X h be L a connected component of the fixed-point set of h, and let T X | X h = Vλ , λ ∈ Q ∩ [0, 1) be a decomposition into direct sum, such that h acts on Vλ as the multiplication P by e2π iλ . Let F(h, X h ⊂ X ) = λ λ(h) be the fermionic shift (see [7], [38]), and let O Vh,X h ⊆X := (3• V0∗ yq k−1 ) ⊗ (3• V0 y −1 q k ) ⊗ (Sym• V0∗ q k ) ⊗ (Sym• V0 q k ) k≥1

⊗

hO (3• Vλ∗ yq k−1+λ(h) ) ⊗ (3• Vλ y −1 q k−λ(h) ) λ6=0

⊗ (Sym

•

i .

Vλ∗ q k−1+λ(h) ) ⊗ (Sym• Vλ q k−λ(h) )

Then we define (see Section 4) Ellorb (X, G; y, q) := y − dim X/2

X

y F(h,X

h ⊆X )

{h},X h

X 1 L(g, Vh,X h ⊆X ), |C(h)| g∈C(h)

where {h} is a conjugacy class in G, C(h) is the centralizer of h, and L(g, Vh,X h ⊆X ) = P i i i (−1) tr(g, H (Vh,X h ⊆X )) is the holomorphic Lefschetz number. Using the Atiyah-Singer holomorphic Lefschetz theorem, the orbifold elliptic genus can be rewritten as follows. For a pair g, h ∈ G of commuting elements, let X g,h be a connected component of the set of points in X fixed by both g and h, let xλ be the Chern roots of a subbundle Vλ of T X | X g,h on which both g and h act via the multiplication by exp(2πiλ(g)) and exp(2πiλ(h)), respectively, and let 8(g, h, λ, z, τ, x) :=

θ(x/(2πi) + λ(g) − τ λ(h) − z) 2π izλ(h)z e . θ(x/(2πi) + λ(g) − τ λ(h))

Then E orb (X, G; z, τ ) =

1 X  |G|

Y

gh=hg λ(g)=λ(h)=0

xλ

Y

8(g, h, λ, z, τ, xλ )[X g,h ]. (1.5)

λ

This formula generalizes (1.2). (As we mentioned earlier, (1.2) has as a consequence (1.3), as was shown in [28].) For a thus-defined orbifold elliptic genus, we prove the

ELLIPTIC GENERA OF SINGULAR VARIETIES

323

formula of Dijkgraaf, Moore, Verlinde, and Verlinde (1.1). We also show that if X is a Calabi-Yau manifold, then E orb (X, G; z, τ ) is a weak Jacobi form. In Section 5 we conjecture (see Conjecture 5.1) that the two notions of elliptic genera coincide, which would extend the results of [5] and [18]. We prove this conjecture for the toric case and in dimension one. For Calabi-Yau hypersurfaces in Gorenstein toric Fano varieties, the elliptic genus has already been defined in [12], using the description of the cohomology of chiral de Rham complex M SV for such hypersurfaces from [10] and borrowing the definition of elliptic genus via chiral de Rham complex in the nonsingular case: Ell(X ) = y dim X/2 Supertrace H ∗ (M SV (X )) y J [0] q L[0] . Here M SV is the chiral de Rham complex constructed in [32] and J [0] and L[0] are the operators of the N = 2 super-Virasoro algebra acting on H ∗ (M SV (X )). We use the combinatorial description of this genus, proved in [12], and the calculation of [11] to show that it coincides with the singular elliptic genus, up to an explicit normalization factor. We continue to discuss Conjecture 5.1 in Section 6. We show that both notions of elliptic genera are invariant under complex cobordisms of G action. By using the known result about cobordism classes of the action of a cyclic group of prime order p, we prove Conjecture 5.1 for involutions.∗

2. Preliminaries 2.1. Elliptic genus Let X be a compact (almost complex) manifold. For a bundle V on X , we consider the following elements in the ring of formal power series over K (X ): X X St (V ) = S i (V )t i , 3t (V ) = 3i (V )t i , i

i

where S i (resp., 3i ) is the ith symmetric (resp., exterior) power of V . Let TX (resp., T¯X ) be the tangent (resp., cotangent) bundle. The elliptic genus of X can be defined as Z Ell(X ; y, q) = ch(E L L y,q ) td(X ), X

where
E L L y,q := y −d/2 ⊗n≥1 3−yq n−1 T¯X ⊗ 3−y −1 q n TX ⊗ Sq n T¯X ⊗ Sq n TX . ∗A

proof of Conjecture 5.1 is given in our paper [13].

324

BORISOV and LIBGOBER

Clearly, this is an invariant of the cobordism class of X , and, moreover, E L L y,q (X × Y ) = E L L y,q (X ) · Ell y,q (Y ); that is, the elliptic genus is a genus in the sense of [27]. If xi are the Chern roots of X , that is, for the total Chern class, we have c(X ) = Q l (1 + xl ); then Z Y θ(xl /(2πi) − z, τ ) Ell(X ; y, q) = xl , (2.1) θ(xl /(2πi), τ ) X l

where q = e2πiτ and y = e2π iz . In (2.1) θ(z, τ ) = q

1/8

(2 sin π z)

l=∞ Y

(1 − q )

l=1

l

l=∞ Y

(1 − q l e2π iz )(1 − q l e−2π iz )

l=1

is the Jacobi theta function (see [14]). In other words, (2.1) is the genus corresponding to the series Q(x) =
x θ(x/(2πi) − z, τ )/θ(x/(2πi), τ ) (see [27]). It is not normalized in the sense that Q(0) = (1/(2πi))(θ(−z, τ )/θ 0 (0, τ )) 6= 1. It is often convinient to use the normalized version of the elliptic genus: Z Y xl θ(xl /(2πi) − z, τ )θ 0 (0, τ ) Ell(X ; y, q) = . (2.2) 2πi θ(xl /(2πi), τ )θ (−z, τ ) X l

For q = 0 we have Ell(X ; y, q = 0) = y −d/2 χ−y (X ), where X p χ y (X ) = (−1)q dim H q (X,  X )y p p,q

is Hirzebruch χ y -genus (see [27]). In particular, Ell(X ; y = 1, q = 0) is the topological Euler characteristic, (−1)d/2 Ell(X ; y = −1, q = 0) is the signature, and so on. If X is a Calabi-Yau, that is, if K X ∼ 0, then Ell(X ; y, q) is a weak Jacobi form. Recall (see [20], [25]) that a weak Jacobi form of weight k ∈ Z and index r ∈ Z/2 is a function on H × C that satisfies  aτ + b z  2 φ , = (cτ + d)k e2πi(r cz /(cτ +d)) φ(τ, z), cτ + d cτ + d φ(τ, z + mτ + n) = (−1)2r (λ+µ) e−2πir (m τ +2mz) φ(τ, z) P and has a Fourier expansion l,m cm,l y l q m with nonnegative m. 2

2.2. Log-terminal singularities We recall basic definitions related to singular varieties. Let Z be a normal irreducible projective variety. The Q-Weil (resp., Q-Cartier) divisor is a linear combination with rational coefficients of codimension one subvarieties (resp., Cartier divisors) on Z .

ELLIPTIC GENERA OF SINGULAR VARIETIES

325

The canonical divisor K Z of Z is a Weil divisor div(s), where s = d f 1 ∧ · · · ∧ d f dim Z ( f i are meromorphic functions) is a nonzero rational differential on Z . We call Z Gorenstein (resp., Q-Gorenstein) if K Z is Cartier (resp., Q-Cartier). A resolution of singularities of a variety Z is a proper birational morphism f : Y → Z , where Y is smooth. Definition 2.1 An IH-small resolution of Z is a regular map Y → Z such that for every i ≥ 1 the set of points z ∈ Z such that dim( f −1 (z)) ≥ i has codimension greater than 2i in Z (see [9]). Definition 2.2 Z has at worst log-terminal singularities if the following two conditions hold. (i) Z is Q-Gorenstein. (ii) For a resolution f : X → Z whose exceptional set is a divisor with simple normal crossings, one has αi > −1 for all i in the relation K X = f ∗ K Z + P αi E i . A well-known result of birational geometry (see, e.g., [15]) states that for any resolution of a log-terminal variety Z , the coefficients αi (called discrepancies) are bigger than (−1). A similar definition of log-terminality exists for pairs (Z , D), where D is a Q-Weil divisor on a normal variety Z such that (K Z + D) is Q-Cartier. 2.3. G-bundles Let X be a complex manifold, and let G be a finite group of holomorphic transformations acting on X . Let V be a holomorphic G-bundle on X ; that is, the action of G on X is extended to the action on V . The holomorphic Lefschetz number of g ∈ G is X
L(g, V ) = (−1)i tr g, H i (X, V ) . i

Let V G be the sheaf whose sections over open sets are the G-invariants of the sections of V . We have (see [26], spectral sequences degenerate due to finiteness of G) χ(V G ) =

1 X L(g, V ). |G| g∈G

The Lefschetz numbers are given by the data around the fixed-point sets (see [2]) as follows. Let N g be the normal bundle to the fixed-point set X g of g, and let N g ∗ be its dual. In the case when the action of G on a space Y is trivial, we have K G (Y ) = K (Y ) ⊗ R(G) (see [2]), and hence one can define W (g) ∈ K (Y ) corresponding to

326

BORISOV and LIBGOBER

W ∈ K G (Y ). In this notation, L(g, V ) =

ch V | X g (g) td(TX g ) g [X ]. ch 3−1 (N g )∗ (g)

(2.3)

For g ∈ G, the normal bundle N X g to the fixed-point set X g can be decomposed into L the direct sum N X g = i N (θi ), θi ∈ Q, where each N (θi ) is the subbundle on which g acts as multiplication by e2πiθi . If xθi , j are the Chern roots of N (θi ), that is, Q if c(N (θi )) = j (1 + xθi , j ), then (2.3) can be rewritten as ch(V | X g ) td(X g )[X g ]. −x −θ j i, j ) i, j (1 − e

L(g, V ) = Q

3. Singular elliptic genus In this section we define singular elliptic genus for a large class of singular varieties and more generally for pairs consisting of a variety and a Q-Cartier divisor on it. This is by far the most general definition of the elliptic genus for singular varieties constructed to date. All varieties are assumed to be proper over Spec(C). Definition 3.1 Let Z be a Q-Gorenstein variety with log-terminal singularities, and let π : Y → Z P be a desingularization of Z whose exceptional divisor E = k E k has simple normal crossings. The discrepancies αk of the components E k are determined by the formula X KY = π ∗ K Z + αk E k . k

Q We introduce Chern roots yl of Y by c(T Y ) = l (1 + yl ) and define cohomology classes ek := c1 (E k ). The singular elliptic genus of Z is then defined as a function of two variables z, τ given by Z Y (yl /(2πi))θ(yl /(2πi) − z)θ 0 (0)  c EllY (Z ; z, τ ) := θ(−z)θ(yl /(2πi)) Y l Y θ(e /(2πi) − (α + 1)z)θ(−z)  k k × , θ(ek /(2πi) − z)θ(−(αk + 1)z) k

where θ(z, τ ) is the Jacobi theta function (see [14]). We often suppress the τ dependence in our formulas. c to be a function of y = e2π iz and q = We usually abuse notation and consider Ell 2π iτ e . Strictly speaking, this function is multivalued because rational powers of y may occur. The key result of this section is the following theorem.

ELLIPTIC GENERA OF SINGULAR VARIETIES

327

THEOREM 3.2 c Y (Z ; y, q) does not depend on the choice of desingularization The above-defined Ell c ; y, q). Y and therefore defines an invariant of Z , which we denote simply by Ell(Z

Proof Because of the weak factorization theorem of [1], it suffices to show that c Y (Z ; y, q) = Ell c ˜ (Z ; y, q) when Y˜ is obtained from Y by a blow-up along a Ell Y nonsingular subvariety X . We remark that the algorithm of [1] is compatible with the normal crossing condition (see [1, Theorem 0.3.1]), so we may assume that X has normal crossings with the components of the exceptional divisor of π : Y → Z . We use the notation of Fulton [23] for the blow-up diagram j X˜ −−−−→   gy

Y˜  f y

X −−−−→ Y i

where X˜ is the exceptional divisor of the blow-down morphism. We also have π : Y → Z and π ◦ f : Y˜ → Z . The discrepancies of the exceptional divisors of these morphisms are related by X KY = π ∗ K Z + αk E k , k

K Y˜ = f π K Z + ∗ ∗

X k

αk E k0 +

X

 αk βk + r − 1 X˜ ,

k

where βk is the multiplicity of E k along X and r is the codimension of X in Y . We use for a while the following technical assumption: The normal bundle N to X inside Y is a pullback under i of some rank r bundle M on Y.

(3.1)

We have the following exact sequences of coherent sheaves on Y˜ (see [23, Section 15.4]): 0 → T Y˜ → f ∗ T Y → j∗ F → 0, 0 → j∗ O X˜ (−1) → j∗ g ∗ i ∗ M → j∗ F → 0, 0 → O → O ( X˜ ) → j∗ O X˜ (−1) → 0, 0 → f ∗ M(− X˜ ) → f ∗ (M) → j∗ g ∗ i ∗ M → 0.

328

BORISOV and LIBGOBER

Here F is the tautological quotient bundle on X˜ . This implies c(T Y˜ ) = c( f ∗ T Y ) · (1 + x) ˜ ·

Y (1 + f ∗ m l − x) ˜ , (1 + f ∗ m l ) l

where c(M) = Therefore,

l (1 + m i )

Q

and x˜ = c1 (O ( X˜ )). Note also that c1 (E k0 ) = f ∗ ek − βk x. ˜

c ˜ (Z ; y, q) Ell Y Z Y ∗ ( f yl /(2πi))θ( f ∗ yl /(2πi) − z)θ 0 (0)  = θ(−z)θ( f ∗ yl /(2πi)) Y˜ l  (x/(2πi))θ ˜ (x/(2πi) ˜ − z)θ 0 (0)  × θ (−z)θ(x/(2πi)) ˜ Y θ(( f ∗ m − x)/(2πi) ˜ − z)(( f ∗ m l − x)/(2πi))θ ˜ ( f ∗ m l /(2πi))  l × θ (( f ∗ m l − x)/(2π ˜ i))( f ∗ m l /(2π i))θ( f ∗ m l /(2πi) − z) l Y θ(( f ∗ e − β x)/(2πi) − (αk + 1)z)θ(−z)  k k˜ × θ (( f ∗ ek − βk x)/(2πi) ˜ − z)θ(−(αk + 1)z) k  θ(x/(2πi) ˜ − (α X˜ + 1)z)θ(−z)  × , θ (x/(2πi) ˜ − z)θ(−(α X˜ + 1)z) P where αx˜ = r − 1 + k αk βk . R R P We now use Y˜ a = Y f ∗ (a). We write the Taylor expansion n Rn (y, q)x˜ n of R the expression under Y˜ in the above identity. Observe that f ∗ R0 (y, q) is exactly the c Y (Z ; y, q); so we need to show that the contribution class in A(Y ) whose integral is Ell of the rest of the terms vanishes. Notice that f ∗ x˜ n = 0 for 1 ≤ n ≤ r − 1 and that P f ∗ x˜ r +n = i ∗ (sn (i ∗ M))(−1)n+r −1 , where n≥0 sn t n is the Segre polynomial of a vector bundle (see [23]). Hence, one needs to calculate Z X i ∗ sn (i ∗ M)(−1)n+r −1 Y n≥0

hY (y /(2πi))θ(y /(2πi) − z)θ 0 (0)  l l × (Coeff. at t r +n ) θ(−z)θ(yl /(2πi)) l  (t/(2πi))θ (t/(2πi) − (α + 1)z)θ 0 (0)  X˜ × θ(t/(2πi))θ(−(α X˜ + 1)z) Y θ((m − t)/(2π i) − z)((m − t)/(2πi))θ(m /(2π i))  l l l × θ((m l − t)/(2πi))(m l /(2πi))θ(m l /(2πi) − z) l Y θ((e − β t)/(2πi) − (α + 1)z)θ (−z) i k k k × . (3.2) θ((ek − βk t)/(2πi) − z)θ(−(αk + 1)z) k

ELLIPTIC GENERA OF SINGULAR VARIETIES

329

We denote nl = i ∗ m l , f k = i ∗ ek and use the fact that X

tr l (t − n l )

sn (i ∗ M)(−1)n t −n = Q

n≥0

to rewrite (3.2) as Z hY (x /(2π i))θ(x /(2πi) − z)θ 0 (0)  l l const. (Coeff. at t −1 ) θ(−z)θ(x l /(2πi)) X l  θ (t/(2πi) − (α + 1)z)θ 0 (0)  X˜ × θ(t/(2πi))θ(−(α X˜ + 1)z) Y  θ((nl − t)/(2πi) − z)θ(nl /(2πi)) × θ((nl − t)/(2πi))(nl /(2πi))θ(nl /(2πi) − z) l Y θ(( f − β t)/(2πi) − (α + 1)z)θ (−z) i k k k × . (3.3) θ(( f k − βk t)/(2πi) − z)θ (−(αk + 1)z) k

Here we denote c(T X ) = l (1 + xl ) and use c(T X ) = i ∗ c(T Y )/i ∗ c(M). To show that (3.3) is zero, observe that the function whose coefficient at t −1 is measured is elliptic in t. Really, t → t + 2π i obviously keeps it unchanged, and t → t + 2πiτ P does not change it because α X˜ = k αk βk + r − 1. Here we have used the fact that none of the α’s is equal to (−1), which follows from the condition that Z is logterminal (see, e.g., [15]). It remains to show that t = 0 is the only pole of this function up to the lattice 2π i(Z + Zτ ), such that the residue is zero. To do so, observe that the normal crossing condition implies βk ∈ {0, 1}, and, moreover, whenever βk = 1, the corresponding factor θ (( f k − t)/(2πi) − z) in the denominator of the last product is offset by a factor θ((nl − t)/(2πi) − z) in the numerator of the second product. We now get rid of assumption (3.1). Indeed, it is easy to see that the difference c Y (Z ; y, q) and Ell c ˜ (Z ; y, q) can be written as a degree of an element of between Ell Y ˜ A( X ) which is preserved when one deforms i : X → Y to the embedding of X into the normal cone for which assumption (3.1) is satisfied. Q

We have not significantly used the log-terminality condition, except for the fact that we did not have to divide by θ(0 · z). We now extend our definition of the singular elliptic genus to the category of pairs that consist of an algebraic variety and a Q-Cartier divisor on it. To avoid (−1) discrepancies, we assume that the pair is Kawamata logterminal. Definition 3.3 Let Z be a projective variety, and let D be an arbitrary Q-Weil divisor such that K Z + D is a Q-Cartier divisor on Z . Let π : Y → Z be a desingularization of Z . We denote

330

BORISOV and LIBGOBER

P by E = k E k the exceptional divisor of π plus the sum of the proper preimages of the components of D, and we assume that it has simple normal crossings. The discrepancies αk of the components E k are determined by the formula X K Y = π ∗ (K Z + D) + αk E k k

and the requirement that the discrepancy of the proper transform of a component of D be the opposite of the coefficient of D at that component. In what follows, we assume that (Z , D) is a Kawamata log-terminal pair, which means that all discrepancies are greater than (−1). Q We introduce Chern roots yl of Y by c(T Y ) = l (1 + yl ) and define Z Y (yl /(2πi))θ(yl /(2πi) − z)θ 0 (0)  c EllY (Z , D; y, q) := θ(−z)θ(yl /(2πi)) Y l Y θ(e /(2πi) − (α + 1)z)θ(−z)  k k × , θ(ek /(2πi) − z)θ(−(αk + 1)z) k

where, as usual, y = c1 (O (E k )).

e2πiz ,

q =

e2π iτ ,

the τ -dependence is suppressed, and ek =

3.4 The above-defined elliptic genus does not depend on the choice of the desingularizac , D; y, q). tion π : Y → Z . We therefore denote it simply by Ell(Z THEOREM

Proof Any two resolutions of singularities of Z can be connected by a sequence of blow-ups and blow-downs, and the argument of Theorem 3.2 works. Kawamata log-terminality implies that all discrepancies on all intermediate varieties are different from (−1). PROPOSITION 3.5 The elliptic genera of two different crepant resolutions of a Gorenstein projective variety coincide.

Proof We show that the elliptic genus of a crepant resolution Y of a variety X equals the singular elliptic genus of X . If the exceptional set of the morphism π : Y → X is a divisor with simple normal crossings, then it is enough to observe that in Definition 3.1 the second product is trivial. In general, we can further blow up Y to get µ : Z → Y so that the exceptional sets of µ and π ◦ µ : Z → X are divisors with simple normal crossings. Then the singular elliptic genera of Y and X calculated via Z are given by the same formula because the discrepancies coincide.

ELLIPTIC GENERA OF SINGULAR VARIETIES

331

Remark 3.6 In particular, the above proposition shows that the statement of [35, Theorem 8.1] can be extended to the full elliptic genus. The following proposition shows that when q → 0, we recover a formula for χ y genus of (Z , D) which follows from [5]. PROPOSITION 3.7 Let (Z , D) be a Kawamata log-terminal pair. Then

c , D; u, q = 0) = (u −1/2 − u 1/2 )dim Z E st (Z , D; u, 1), Ell(Z where E st is defined as in [5]. Proof To avoid confusion, we immediately remark that the second arguments in the singular elliptic genus and in Batyrev’s E-function have drastically different meanings. The definition of E st (Z , D) in [5] could be stated as  X Y  uv − 1 E st (Z , D; u, v) := E(E J ; u, v) − 1 , α j +1 − 1 uv J ⊂I j∈J P where i∈I αi E i is the exceptional divisor of a resolution Y → Z together with proper preimages of the components of D and is assumed to have normal crossings. Polynomials E(E J ; u, v) are defined in terms of mixed Hodge structure on the cohoT mology of E J (see [5]). Subvariety E J is j∈J E j , and the sum includes the empty subset J . For each J , Z dim YE J (1 − u e−xi,J )xi,J , E st (E J ; u, 1) = 1 − e−xi,J E j i=1 Q where c(T E J ) = i (1 + xi , J ). The adjunction formula for complete intersections yields
Y
c(T E J ) = i ∗J c(T Y ) / 1 + i ∗J c1 (E j ) , j∈J

where i J : E J → Y is the closed embedding. We then obtain ∗ Z dim YY (1 − u e−i ∗J xi )i ∗ xi Y 1 − e−i J e j J E(E J ; u, 1) = ∗ −i ∗J e j ∗ 1 − e−i J xi )i J e j E j i=1 j∈J (1 − u e Z =

dim YY Y i=1

(1 − u e−xi )xi Y 1 − e−e j , 1 − e−xi 1 − u e−e j j∈J

332

BORISOV and LIBGOBER

where c(T Y ) =

i (1 + x i ).

Q

E st (Z , D; u, 1) =

Z

When we plug this result into Batyrev’s formula, we get

dim YY Y i=1

Z =

dim YY Y i=1

(1 − u e−xi )xi Y (u − u α j +1 )(1 − e−e j )  1 + 1 − e−xi (u α j +1 − 1)(1 − u e−e j ) j∈I (1 − u e−xi )xi Y (u − 1)(1 − u α j +1 e−e j ) 1 − e−xi (u α j +1 − 1)(1 − u e−e j ) j∈I

c , D; u, q). = (u −1/2 − u 1/2 )− dim Z lim Ell(Z q→0

The following simple proposition establishes the modular properties of the singular elliptic genus in the Calabi-Yau case. PROPOSITION 3.8 Let (Z , D) be a Kawamata log-terminal pair which is also a Calabi-Yau pair in the sense that K Z + D is zero as a Q-Cartier divisor. Then the singular elliptic genus c , D; y, q) has the transformation properties of the Jacobi form of weight dim Z Ell(Z and index zero for the subgroup of the full Jacobi group generated by

(z, τ ) → (z + n, τ ),

(z, τ ) → (z + nτ, τ ),  z −1  (z, τ ) → , , τ τ

(z, τ ) → (z, τ + 1),

where n is the least common denominator of the discrepancies. Proof The transformation properties of θ(z, τ ) under (z, τ ) → (z + 1, τ ) and (z, τ ) → (z + τ, τ ) together with Calabi-Yau condition X KY = αk E k k

assure that c , D; z + n, τ ) = Ell(Z c , D; z + nτ, τ ) = Ell(Z c , D; z, τ ). Ell(Z We need here the fact that nαk ∈ Z. Similarly, the transformation properties of θ under (z, τ ) → (z, τ + 1) show that c , D; z, τ + 1) = Ell(Z c , D; z, τ ). Ell(Z It remains to investigate what happens under (z, τ ) → (z/τ, −1/τ ). For this, one considers the change (ek , yl ) → (ek /τ, yl /τ ) in the formula of Definition 3.3. A

ELLIPTIC GENERA OF SINGULAR VARIETIES

333

rather lengthy but straightforward calculation, similar to that of [12, Theorem 2.2], shows that   c Z , D; z , − 1 = τ dim Z Ell(Z c , D; z, τ ). Ell τ τ Another application of our techniques is the following theorem, which complements similar results for Hodge numbers of Calabi-Yau manifolds (see, e.g., [4] and [17]). THEOREM 3.9 The elliptic genera of two birationally equivalent Calabi-Yau manifolds coincide. Moreover, the statement is true for smooth projective algebraic manifolds X with n K X ∼ 0 for some n.

Proof Let Z 1 and Z 2 be two birationally equivalent Calabi-Yau manifolds or their generalizations above. Let Y be a desingularization of the closure of the graph of the birational equivalence such that π1,2 : Y → Z 1,2 are regular birational morphisms. Let n be the smallest integer such that n K Z 1,2 is rationally equivalent to zero and therefore has a global section. Global sections of the pluricanonical bundle are birational P invariants, so one can consider the divisor k ak E k of this section on Y . It is easy to P see that for both morphisms π1 and π2 the exceptional divisor is k (ak /n)E k , which we can then assume to have simple normal crossings (perhaps by passing to a new desingularization). Therefore, the elliptic genera of Z 1,2 are calculated on Y using the same discrepancies. Remark 3.10 It is interesting to compare the results of this section with the work of Totaro in [35], where he tried to see which Chern numbers can be meaningfully defined for singular varieties. For varieties that admit IH-small resolutions, the singular elliptic genus provides the maximum possible collection of such numbers. Totaro has shown that every flop-invariant Chern number comes from the elliptic genus, and he obtained partial results in the opposite direction by means of intersection cohomology. In general, coefficients of the singular elliptic genus of Z at y k q l provide analogs of Chern numbers of singular varieties in the following sense. (1) They are the invariants of the isomorphism class of singular spaces. (2) For manifolds, these invariants are the usual Chern numbers (i.e., linear comP binations of ci1 (X ) · · · ci N (X )[X ], where k=N k=1 i k = dim X and [X ] is the fundamental class of a manifold X ). (3) These invariants are unchanged under small resolutions. In fact, for singular varieties, elliptic genera may contain more information than

334

BORISOV and LIBGOBER

in the nonsingular case. For varieties with non-Gorenstein singularities, the singular elliptic genus may depend on rational powers of y. Moreover, there exist examples of Gorenstein varieties whose elliptic genera do not lie in the span of elliptic genera of nonsingular varieties. This can be seen already at the level of the χ y genus (see [3] for an example of a variety with Gorenstein canonical singularities whose E-function is not a polynomial). We hope that elliptic genera of singular varieties can be interpreted as nontrivial invariants of a not-yet-defined cobordism theory of singular spaces. Transformations leaving the singular elliptic genus invariant in such a theory for smooth manifolds should include the usual cobordisms as well as flops. It would be interesting to compare our results with the invariants of Witt spaces studied by P. Siegel; the latter, however, were defined in S O rather than in the complex category (see [9], [34]). Remark 3.11 It is an open question whether the notion of singular elliptic genus can be extended beyond Kawamata log-terminal pairs, and some partial resuts in this direction can be obtained as follows. Assume that some of the discrepancies αk in Definition 3.3 equal (−1). One can try to define singular elliptic genus by continuity. Namely, for any effective Cartier divisor H on Z that contains all singular points of Z , and whose preimage on Y has simple normal crossings with the exceptional divisor and preimage of of D, we calculate c Y (Z , D + H/n; z, τ ) lim Ell n→∞

for each (z, τ ). If such a limit exists and is independent of H , then we call it c Y (Z , D; z, τ ). Notice that if n is sufficiently big, then the discrepancies of all diEll visors E k calculated for the pair (Z , D + H/n) are not equal to (−1). However, we do not know of any necessary or sufficient conditions for the limit to exist. We also cannot prove in general that this limit is independent of Y . In particular, we cannot show in general that two resolutions Y1 and Y2 with no (−1) discrepancies give the same singular elliptic genus because the sequence of blow-ups and blow-downs which connects Y1 to Y2 may have intermediate varieties with (−1) discrepancies. Provided a single divisor H can be chosen to satisfy the normal crossing condition for both resolutions, one gets c Y1 (Z , D; z, τ ) = lim Ell c Y1 (Z , D + H/n; z, τ ) Ell n→∞

c Y2 (Z , D + H/n; z, τ ) = Ell c Y2 (Z , D; z, τ ), = lim Ell n→∞

where the middle identity follows from the argument of Theorem 3.2 since for large n all intermediate discrepancies will be different from (−1). A similar approach allows one to extend the definition of elliptic genus to arbitrary Q-divisors D on a c , (n + 1)D/n). log-terminal variety Z by looking at limn→∞ Ell(Z

ELLIPTIC GENERA OF SINGULAR VARIETIES

335

4. Orbifold elliptic genus and DMVV formula In this section we define orbifold elliptic genus, which we conjecture to equal the singular elliptic genus of Section 3. We delay the comparison of these two genera until Section 5. Instead, the goal of this section is to show how this definition of orbifold elliptic genus allows one to recover the formula of [19] whose derivation was based partly on heuristic string-theoretic arguments. Our definition of elliptic genus is inspired by the calculations of [10]. Definition 4.1 Let X be a smooth algebraic variety acted upon by a finite group G. We assume that the subgroup of elements of G acting trivially on X contains only the identity. We define the following function of two variables, which we call the orbifold elliptic genus of X/G: Ellorb (X, G; y, q) := y − dim X/2

X

y F(h,X

h ⊆X )

{h},X h

X 1 L(g, Vh,X h ⊆X ), |C(h)| g∈C(h)

where F(h, X h ⊆ X ) is the fermionic shift (see [38], [7]) and Vh,X h ⊆X is a vector bundle over X h defined as follows. Let T X | X h decompose into eigensheaves for h as   M (4.1) V0 ⊕ Vλ . λ:hhi→Q/Z

We lift λ(h) to a rational number in [0, 1). Then Vh,X h ⊆X is defined as O Vh,X h ⊆X := (3• V0∗ yq k−1 ) ⊗ (3• V0 y −1 q k ) ⊗ (Sym• V0∗ q k ) ⊗ (Sym• V0 q k ) k≥1

⊗

hO (3• Vλ∗ yq k−1+λ(h) ) ⊗ (3• Vλ y −1 q k−λ(h) ) λ6=0

i ⊗ (Sym• Vλ∗ q k−1+λ(h) ) ⊗ (Sym• Vλ q k−λ(h) ) . Remark 4.2 Another way to state this definition is Ellorb (X, G; y, q) := y − dim X/2

X

y F(h,X

h ⊆X )


C(h) χ H • (Vh,X h ⊆X ) .

{h},X h

THEOREM 4.3 Let X and G be as above, and let X g,h be the set of fixed points of a pair of commuting L elements g, h ∈ G. Let T X | X g,h = Wλ be the decomposition (refinement of (4.1))

336

BORISOV and LIBGOBER

of the restriction on X g,h of the tangent bundle into the direct sum of line bundles on which g (resp., h) acts as multiplication by e2π iλ(g) (resp., e2π iλ(h) ). Denote by xλ the Chern roots of the bundle Wλ . (1) We have Ellorb (X, G) =

1 |G| ×

X



Y

xλ



g,h,gh=hg λ(g)=λ(h)=0

Y θ(τ, xλ /(2πi) + λ(g) − τ λ(h) − z) λ

θ(τ, xλ /(2πi) + λ(g) − τ λ(h))

e2π iλ(h)z [X g,h ].

(2) Let X be a Calabi-Yau of dimension d, such that H 0 (X, K X ) = C. Denote by n the order of G in AutH 0 (X, K X ). Then Ellorb (X, G) is a weak Jacobi form of weight zero and index d/2 with respect to the subgroup of the Jacobi group 0 J generated by transformations (z, τ ) → (z + n, τ ), (z, τ ) → (z, τ + 1),

(z, τ ) → (z + nτ, τ ), z 1 (z, τ ) → ,− . τ τ

In particular, if the action preserves holomorphic volume, then Ellorb (X, G) is a weak Jacobi form of weight zero and index d/2 for the full Jacobi group. Proof We replace the contribution of each conjugacy class by an average contribution of its elements to obtain Ellorb (X, G) =

1 − dim X/2 X F(h,X h ⊂X ) y y L(g, Vh,X h ⊂X ). |G| gh=hg

Using holomorphic Lefschetz theorem, we obtain Ellorb (X, G) =

1 − dim X/2 X F(h,X h ⊂X ) ch(Vh,X h ⊂X | X g,h )(g) td(TX g,h )[X g,h ] y y , g |G| ch3−1 (N h )∗ (g) gh=hg

g

X

where N X h is the normal bundle to X g,h in X h . An explicit calculation of the Chern

ELLIPTIC GENERA OF SINGULAR VARIETIES

337

and Todd classes then yields Ellorb (X, G) =

1 X F(h,X h ⊂X )−dim X/2  y |G| gh=hg

Y

xλ



λ(g)=λ(h)=0

Y (1 − yq k−1+λ(h) e−xλ −2π iλ(g) )(1 − y −1 q k−λ(h) e xλ +2πiλ(g) ) × (1 − q k−1+λ(h) e−xλ −2πiλ(g) )(1 − q k−λ(h) e xλ +2π iλ(g) ) k≥1,λ  Y 1 X  = xλ |G| gh=hg λ(h)=λ(g)=0

×

Y θ (xλ /(2πi) + λ(g) − τ λ(h) − z) λ

θ (xλ /(2πi) + λ(g) − τ λ(h))

e2π izλ(h) [X g,h ],

which proves the first part of the theorem. To verify the modular property, we denote 8(g, h, λ, z, τ, x) :=

θ(x/(2πi) + λ(g) − τ λ(h) − z) 2πizλ(h) e , θ(x/(2πi) + λ(g) − τ λ(h))

where λ is a character of the subgroup of G generated by g and h. Then E orb (z, τ ) =

1 X  |G|

Y

xλ

Y

gh=hg λ(g)=λ(h)=0

8(g, h, z, τ, xλ )[X g,h ],

(4.2)

λ

where we suppress (X, G) from the notation for the sake of brevity. We have 8(g, h, λ, z + 1, τ, x) = − e2π iλ(h) · 8(g, h, λ, z, τ, x), and hence Ellorb (z + n, τ ) = (−1)dn Ellorb (z, τ ) since by assumption n · It is clear that

P

λ(h) ∈ Z.

8(g, h, λ, z, τ + 1, x) = 8(gh −1 , h, λ, z, τ, x), and hence Ellorb (z, τ + 1) = Ellorb (z, τ ). We have 8(g, h, λ, z + nτ, τ, x) = (−1)n e−2π inz−πin

2τ

enx+2π inλ(g) · 8(g, h, λ, z, τ, x),

and hence Ellorb (z + nτ, τ ) = (−1)dn e−2π idnz−πidn

2τ

Ellorb (z, τ )

338

BORISOV and LIBGOBER

since X is a Calabi-Yau and nλ(g) ∈ Z. Finally,  1 x z 8 g, h, λ, , − , τ τ τ θ (−z/τ + x/(2πiτ ) + λ(g) + λ(h)/τ , −1/τ ) 2πizλ(h)/τ = e θ (xλ /(2πi) + λ(g) + λ(h)/τ , −1/τ ) = eπ iz /τ −2π iz/τ (x/(2π i)+λ(g)τ +λ(h)) θ (−z + x/(2π i) + λ(g)τ + λ(h), τ ) 2πizλ(h)/τ × e θ (x/(2πi) + λ(g)τ + λ(h), τ ) θ(−z + x/(2πi) + λ(g)τ + λ(h), τ ) 2πiz(−λ(g)) 2 = eπ iz /τ −zx/τ · e θ(x/(2πi) + λ(g)τ + λ(h), τ ) 2

= eπ iz

2 /τ −zx/τ

· 8(h, g −1 , λ, z, τ, x).

Then the Jacobi transformation properties follow easily from (4.2), similarly to [12, proof of Theorem 2.2]. It is straightforward to see from (4.2) that the orbifold elliptic genus is holomorphic and has the Fourier expansion with nonnegative powers of q. We apply our definition of the orbifold elliptic genus to symmetric products of a smooth variety. This gives a mathematical justification of the physical calculation performed in [19]. More precisely, we calculate the generating function for the orbifold elliptic genera introduced above for the action of the symmetric groups. To a certain extent, our calculation follows [19], but we now have precise mathematical definitions. THEOREM 4.4 P Let X be a smooth variety with elliptic genus m,l c(m, l)y l q m , where the elliptic genus is normalized as in [19] and [12]. Then

X n≥0

p n Ellorb (X n , 6n ; y, q) =

∞ Y Y i=1 l,m

1 (1 −

pi y l q m )c(mi,l)

.

We start with the following lemma, essentially contained in [19, Section 2.2], which we include only for completeness. 4.5 Let V = Veven ⊕ Vodd be a supersymmetric space, and let A and B be two commuting operators preserving parity decomposition of V , such that B has only nonnegative integer eigenvalues. We assume that V splits into a direct sum of eigenspaces Vm of LEMMA

ELLIPTIC GENERA OF SINGULAR VARIETIES

339

the operator B and that each Vm is finite-dimensional. Define χ (V )(y, q) = SupertraceV y A q B := trVeven (y A q B ) − trVodd (y A q B ) X = d(m, l)q m y l , m,l

where d(m, l) is the superdimension of the space Vm,l = {v ∈ V |Av = lv, Bv = mv}. The operators A and B act on the space of invariants of the symmetric group N acting on V ⊗ and X

p N SupertraceSym N (V ) y A q B =

N

1

Y m,l

(1 −

pq m y l )d(m,l)

,

where the right-hand side is expanded as a power series in q and p. Proof It is easy to see that it is enough to check the lemma for a one-dimensional space V = Vm,l . If V is even, then X X p N SupertraceSym N (V ) y A q B = p N y Nl q N m = (1 − pq m y l )−superdimV . N

N ≥0

If V is odd, then X p N SupertraceSym N (V ) y A q B = 1 − py l q m = (1 − pq m y l )−superdimV . N

Proof of Theorem 4.4 We observe that for a fixed k the conjugacy classes of 6k are indexed by the numbers ai of cycles of length i in the permutation. For each h ∈ 6k , the fixed-point set (X k )h consists of the Cartesian products of several copies of X , one for each cycle. For a cycle of length i, the corresponding X is embedded into X i . The centralizer group Q is a semidirect product of its normal subgroup i (Z/iZ)ai , which acts by cyclic Q permutations inside cycles of h, and the product of symmetric groups i 6ai , which act by permuting cycles of the same length.

340

BORISOV and LIBGOBER

Our definition of the elliptic genus then gives X p n Ellorb (X n , 6n ; y, q) n≥0

X

=

pa1 +2a2 +···+nan y −(dim X /2)(a1 +2a2 +···+nan )

a1 ,a2 ,...,an n Y ai F(i−cycle,X ⊆X i )

y

×

N

χ (H • (Vi−cycle,X ⊆X i )

ai 6ai o(Z/iZ)ai

)




i=1

=

∞ Y


i χ Sym• ( pi y −i dim X/2+F(i−cycle,X ⊆X ) H • (Vi−cycle,X ⊆X i )Z/iZ ) . (4.3)

i=1

The symbol Sym should be interpreted here as the supersymmetric product where the cohomology of Vh,X h ⊆X is given parity by the sum of the cohomology number and the parity of the exterior algebras. We now calculate
i χi (y, q) = χ pi y −i dim X/2+F(i−cycle,X ⊆X ) H • (Vi−cycle,X ⊆X i )Z/iZ . We denote the i-cycle by h and observe that M T X i |X =

T X j.

j=0,...,i−1;λ(h)= j/i

This implies F(h, X ⊆ X i ) = dim X us to write

Pi−1

j=0

j/i = dim X ((i − 1)/2), which allows

χi (y, q) = pi y − dim X/2 χ

h

O H •( (3• T ∗ yq k−1 ) ⊗ (3• T y −1 q k ) ⊗ (Sym• T ∗ q k ) k≥1

⊗ (Sym• T q k ) ⊗



O

(3• T ∗ yq k−1+ j/i )

j=1,...,i−1

⊗ (3 T y

) ⊗ (Sym• T ∗ q k−1+ j/i ) Z/iZ i ⊗ (Sym• T q k− j/i ) •

i − dim X/2 1

=p y

×

i

i−1 Z X r =0

−1 k− j/i

dim YX

q

xl

X l=1

i−1 YY (1 − yq k−1+m/i ξ mr e−xl )(1 − y −1 q k−m/i ξ −mr exl ) (1 − q k−1+m/i ξ mr e−xl )(1 − q k−m/i ξ −mr exl )

k≥1 m=0

ELLIPTIC GENERA OF SINGULAR VARIETIES

i − dim X/2 1

=p y ×

i−1 Z X

341

dim YX

xl r =0 X l=1 yq ( j−1)/i ξ ( j−1)r e−xl )(1 − i

Y (1 − y −1 q j/i ξ jr exl ) (1 − q ( j−1)/i ξ ( j−1)r e−xl )(1 − q j/i ξ jr exl ) j≥1

= pi 1/i

i−1 X r =0

Ell(X ; y, q 1/i ξ r ) =

X

c(mi, l)y l q m .

m,l

Here we have denoted the primitive ith root of unity by ξ . Now Lemma 4.5 finishes the proof. Remark 4.6 In [37] the authors conjectured an equivariant version of Theorem 4.4. Its proof follows using the same arguments as above. More precisely, we have the following. Let X and G be as above, and let G o 6n be the wreath product consisting of pairs ((g1 , . . . , gn ); σ ), gi ∈ G, σ ∈ 6n , with multiplication ((g1 , . . . , gn ); σ1 ) ·
((h 1 , . . . , h n ); σ2 ) = ((g1 · h σ −1 (1) , . . . , gn · h σ −1 (n) ); σ1 σ2 ) . G o 6n acts in an obvi1 P1 ous way on X n , and if Ellorb (X, G; y, q) = cG (m, l)y l q m , then X

p n Ellorb (X n , G o 6n ; y, q) =

n≥0

∞ Y Y i=1 l,m

1 (1 −

pi y l q m )cG (mi,l)

.

(4.4)

To obtain a proof of this formula, one should make the following changes in the above proof of Theorem 4.4. Using the description of the conjugacy classes in P wreath products (see, e.g., [29]), n≥0 p n Ellorb (X n , G o 6n ; y, q) can be rewritten as the right-hand side of the first row of (4.3) with summation taken over collections {h}, a1 , . . . , an , where ai , as earlier, are positive integers and {h} runs through all conjugacy classes in G. The same transformation used in (4.3) now yields the product over i and {h} of terms in which invariants are taken for the semidirect product of the centralizer of h and Z/iZ with the sheaf V constructed for X h . Finally, each term in this product is the graded dimension of a supersymmetric algebra, which Lemma 4.5 expresses in terms of χi,{h} . A calculation similar to the calculation of χi above identifies χi,{h} with X m,l

c{h} (mi, l)y l q m = y − dim X/2+F(h,X

h ⊆X )

X 1 L(g, Vh,X h ⊆X ) |C(h)| g∈C(h)

(the component of the orbifold elliptic genus corresponding to the conjugacy class {h}). This yields (4.4).

342

BORISOV and LIBGOBER

5. Comparison of different notions of elliptic genera It is natural to ask how the orbifold elliptic genus of X/G is related to its singular elliptic genus. To begin, even in the case |G| = 1, these two genera differ by a normalization factor. In addition, when µ : X → X/G has a ramification divisor P c D = i (νi − 1)Di , one has to compare Ellorb (X, G; y, q) not to Ell(X/G; y, q) but c rather to Ell(X/G, 1 X/G ; y, q), where 1 X/G :=

X νj − 1 µ(D j ) νj j

with the sum taken among representatives D j of the orbits of the action of G on the components of the ramification divisor. CONJECTURE 5.1 Let X be a smooth algebraic variety equipped with an effective action of a finite group G. Then  2πi θ(−z, τ ) dim X c Ellorb (X, G; y, q) = Ell(X/G, 1 X/G ; y, q), θ 0 (0, τ )

where 1 X/G is defined as above. We now present some evidence to support this conjecture. 5.2 Conjecture 5.1 holds in the limit τ → i∞. PROPOSITION

Proof At q = 0, the function Ellorb specializes to E orb (y, 1) of [5]. Then the result of [18] allows one to rewrite it in terms of E st (y, 1), and Proposition 3.7 finishes the proof. PROPOSITION 5.3 Conjecture 5.1 holds in the case when X is a smooth toric variety and G is a subgroup of the big torus of X .

Proof Let 6 be the defining cone of X in the lattice N (see, e.g., [16]). Let n i be the generators of one-dimensional cones of 6. The group G can be identified with N 0 /N , where N 0 is a sublattice of N of finite coindex. Then the variety X/G is given by the same cone 6 in the new lattice N 0 . The map µ : X → X/G has ramification if and only if for some one-dimensional rays of 6 points n i are no longer minimal in the new lattice.

ELLIPTIC GENERA OF SINGULAR VARIETIES

343

Torus-invariant divisors on a toric variety correspond to piecewise linear functions on the fan. It is easy to see that the definition of 1 X/G assures that the piecewise linear function that takes values (−1) on all n i gives the divisor K X/G + 1 X/G . We denote this piecewise linear function by deg. One can show that  2πiθ(−z, τ ) dim X Ellorb (X, G; y, q) = f N 0 ,deg z (q), θ 0 (0, τ ) where f N 0 ,deg z (q) is the function defined in [11]. More explicitly,  X X X f N 0 ,deg z (q) = (−1)codim C a.c. q m·n e2πiz deg(n) , m∈(N 0 )∗ C∈6

n∈C∩N 0

where a.c. means analytic continuation. The proof of this fact is based on explicit ˇ calculation of the Euler characteristics of the bundles V X h ⊆X by means of Cech cohomology. The calculation is very similar to that of [11, Theorem 3.4] and is left to the reader. We remark that the sum over h in Definition 4.1 facilitates the change from N to N 0 , while taking C(h)-invariants is responsible for the switch from N ∗ to its sublattice (N 0 )∗ . Now let Y → X/G be a toric desingularization of X/G given by the subdivision 61 of 6. We denote the codimension one strata of Y by E k and the generators of the corresponding one-dimensional cones of 61 by rk . We also denote the first Chern classes of the corresponding divisors by ek , and we get Z Y (yl /(2πi))θ(yl /(2πi) − z)θ 0 (0)  c Ell(X/G, 1 X/G ; y, q) = θ(−z)θ(yl /(2π i)) Y l Y θ(e /(2πi) − (α + 1)z)θ(−z)  k k × , θ(ek /(2πi) − z)θ(−(αk + 1)z) k Q Q where c(T Y ) = l (1 + yl ) and αk = deg(rk ) − 1. We use c(T Y ) = k (1 + ek ) to c rewrite Ell(X/G, 1 X/G ; y, q) as Z Y (ek /(2πi))θ (ek /(2πi) − deg(rk )z)θ 0 (0)  , θ(− deg(rk )z)θ(ek /(2πi)) Y k

which equals f N 0 ,deg z (q) by [11, Theorem 3.4]. We have used here the fact that f does not change when the fan is subdivided. Remark 5.4 Proposition 5.3 was the main motivation behind our definition of the singular elliptic genus. 5.5 Conjecture 5.1 holds for dim X = 1. PROPOSITION

344

BORISOV and LIBGOBER

Proof Expanding θ functions as (linear) polynomials in cohomology classes, one obtains that the singular genus is equal to (2g − 2)θ 0 (−z)/(2πiθ(−z)) plus the sum of contributions of singular points that depend on the ramification numbers only. Here g is the genus of X/G. For the orbifold genus, one needs to notice that the h = id term gives (2g − 2)θ 0 (−z)/(2πiθ(−z)) plus contributions of points because it is the Euler characteristic of the bundle on the quotient that equals the usual elliptic genus bundle twisted at the ramification points. Since the equality holds in the toric case of the d-fold covering of P1 by P1 , which has two points of ramification (d − 1), the extra terms for the two genera coincide, which finishes the proof. One would also want to compare the singular elliptic genus to the elliptic genus defined for toric varieties and Calabi-Yau hypersurfaces in toric varieties in [12]. It turns out that these definitions agree, up to a normalization. We explain the Calabi-Yau case in more detail and leave the toric case to the reader. We need to recall the combinatorial description of Calabi-Yau hypersurfaces in toric varieties and the previous definition of their elliptic genera. Let M1 and N1 be dual free abelian groups of rank d + 1. Denote by M and N the dual free abelian groups M = M1 ⊕ Z and N = N1 ⊕ Z. Element (0, 1) ∈ M is denoted by deg, and element (0, 1) ∈ N is denoted by deg∗ . There are dual reflexive polytopes 1 ∈ M1 and 1∗ ∈ N1 which give rise to dual cones K ⊂ M and K ∗ ⊂ N . Namely, K is a cone over (1, 1) with vertex at (0, 0) M , and similarly for K ∗ . There is a complete fan 61 on N1 whose one-dimensional cones are generated by some lattice points in 1∗ (in particular, by all vertices). This fan induces the decomposition of the cone K ∗ into subcones, each of which includes deg∗ . Let us denote this decomposition by 6. A generic Calabi-Yau hypersurface X f of the family given by the above combinatorial data is determined by a choice of coefficients f m for each m ∈ (1, 1). The elliptic genus of X f was defined in [12] as the graded Euler characteristic of a certain sheaf of vertex algebras on X f . We do not need to recall the definition of this sheaf in view of the following combinatorial formula for the elliptic genus. PROPOSITION 5.6 The elliptic genus Ell(X f ; y, q) of the Calabi-Yau hypersurface X f , as defined in [12], is given by  X  X ∗ ∗ Ell(X f ; y, q) = y −d/2 a.c. y n·deg−m·deg q m·n+m·deg G(y, q)d+2 , m∈M

n∈K ∗

ELLIPTIC GENERA OF SINGULAR VARIETIES

345

where a.c. stands for analytic continuation and G(y, q) =

Y (1 − yq k−1 )(1 − y −1 q k ) . (1 − q k )2

k≥1

Proof Combine [12, Proposition 4.2] and [12, Definition 5.1]. THEOREM 5.7 The elliptic genus of the Calabi-Yau hypersurface X f of dimension d defined above and its singular elliptic genus are related by the formula  2πiθ(−z, τ ) d c f ; y, q). Ell(X f ; y, q) = Ell(X θ 0 (0, τ )

Proof First of all, observe that y −1/2 G(y, q) =

2πiθ(−z, τ ) , θ 0 (0, τ )

due to the product formulas for θ(z, τ ) and θ 0 (0, τ ) (see [14]). Therefore, we only need to show that  X  X ∗ ∗ c f ; y, q) = Ell(X a.c. y n·deg−m·deg q m·n+m·deg G(y, q)2 . m∈M

n∈K ∗

Denote by deg1 the piecewise linear function on N1 whose value on the generators of the one-dimensional cones of 61 is 1. Notice that K ∗ consists of all points (n 1 , l) ∈ N P P such that l ≥ deg1 (l). In addition, one can replace n∈K . . . by C∈6 (−1)codim 6 . . . to get  X  X ∗ ∗ a.c. y n·deg−m·deg q m·n+m·deg G(y, q)2 m∈M

n∈K ∗

=

X X X

=

X X X

(−1)codim C1 a.c.

X

X

y l−k q m 1 ·n 1 +lk+k G(y, q)2

n 1 ∈C1 l≥deg1 (n 1 )

k∈Z m 1 ∈M C1 ∈61

(−1)codim C1

k∈Z m 1 ∈M C1 ∈61

× a.c.

X

X

y deg1 (n 1 )−k q m 1 ·n 1 +deg1 (n 1 )k+k (1 − yq k )−1 G(y, q)2

n 1 ∈C1 l≥deg1 (n 1 )

= G(y, q)2

X k∈Z

y −k q k f N ,deg z (yq k , q). (1 − yq k ) 1 1

346

BORISOV and LIBGOBER

Let 610 be a refinement of the fan 61 in N1 such that the corresponding toric variety P6 0 is smooth. Coefficients f m define a hypersurface X 0f in P6 0 which is a resolution 1 1 of singularities X f . We denote the codimension one strata of P6 0 by D j , their first 1 Chern classes by d j , and the corresponding generators of one-dimensional cones of 610 by n j . By [11, Theorem 3.4], we get G(y, q)2

X y −k q k f N ,deg z (yq k , q) (1 − yq k ) 1 1 k∈Z

= G(y, q)2

X k∈Z

Y (d j /(2πi))θ (d j /(2πi) − deg1 (n j )(z + kτ ))θ 0 (0)

Z × P6 0

1

= P6 0

j

1

×

X k∈Z

We denote D = we get X

θ(− deg1 (n j )z)θ(n j /(2πi)) G(y, q)2

 P y −k q k k j d j deg1 (n j ) e . (1 − yq k )

deg1 (n 1 )D j and d = c1 (D). Because of [12, Proposition 3.2],

j

G(y, q)2

k∈Z

θ(− deg1 (n j )(z + kτ ))θ(n j /(2πi))

j

Y (d j /(2πi))θ (d j /(2πi) − deg1 (n j )z)θ 0 (0)

Z

P

y −k q k (1 − yq k )

P y −k q k G(ed q, q)G(y, q) k j d j deg1 (n j ) e = (1 − yq k ) G(y −1 ed q, q)

=

2πiθ(d/(2πi))θ(−z, τ ) , θ(d/(2πi) − z)θ 0 (0)

which gives X

 X  ∗ ∗ a.c. y n·deg−m·deg q m·n+m·deg G(y, q)2 n∈K ∗

m∈M

Y (d j /(2πi))θ (d j /(2πi) − deg1 (n j )z)θ 0 (0)

Z =

θ(− deg1 (n j )z)θ(n j /(2πi))  2πiθ(d/(2πi))θ(−z, τ )  × . θ(d/(2πi) − z)θ 0 (0) P6 0

1

j

Observe now that D = π ∗ (−K P61 ), where π : P6 0 → P61 is the resolution induced 1 by the subdivision of the fan. In addition, X f is a zero set of a section of D. Hence, the adjunction formula gives c(TX 0f ) = i ∗ c(T P6 0 )/(1 + i ∗ d), 1

ELLIPTIC GENERA OF SINGULAR VARIETIES

347

where i : X 0f → P6 0 is the embedding. The exceptional divisors of X 0f → X f are 1 D j ∩ X 0f (unless dim π(D j ) = 0), and their discrepancies are equal to deg(n j ) − 1. Then it is easy to see that the above expression is precisely the singular elliptic genus of X f . Remark 5.8 The case of toric varieties is a straightforward application of [11, Theorem 3.4] and is left to the reader. Remark 5.9 The above calculations indicate that for any smooth variety P of dimension d + 1 one can define a weak Jacobi form of weight d and index zero which coincides with the singular elliptic genus of the Calabi-Yau hypersurface in P if P has smooth anticanonical divisors. Otherwise, the formula gives the elliptic genus of the “virtual” Calabi-Yau hypersurface in P. One can also interpret this Jacobi form as an elliptic genus of a (d + 1, 1)-dimensional Calabi-Yau supermanifold 5K X (a canonical line bundle over X , considered as an odd bundle). 6. Cobordism invariance of orbifold elliptic genus We view Ellsing (X/G) and Ellorb (X, G) as invariants of G-action on X and work in the category of stably almost complex manifolds. LEMMA 6.1 A singular elliptic genus is an invariant of a complex G-cobordism.

Proof We consider cobordisms of pairs (X, D) (see [36]), where X is a stably almost complex manifold (i.e., a C ∞ -manifold such that a direct sum of a trivial bundle  with S the differentiable tangent bundle TX admits a complex structure) and D = Di is a finite union of codimension one stably almost complex submanifolds (i.e., TDi ⊕  is a complex subbundle in  ⊕ TX ) satisfying the following normal crossing condition: at each point of Di1 ∩ · · · ∩ Dik , the union of (stabilized by adding trivial bundles) tangent spaces TDi j ⊕  is given in the (stabilized) tangent space to X by l1 · · · lk = 0, where li are linearly independent complex linear forms. A pair (X, D) is cobordant to zero if there exist a C ∞ -manifold Y with a complex structure on the stable tangent S S S bundle and a system of submanifolds E i such that ∂Y = X and ∂ E i = Di . As usual, the disjoint union and product provide the ring structure on cobordism classes. Notice that the numbers ci1 ∪ · · · ∪ cik ∪ [D j1 ] ∪ · · · ∪ [D jk ]([X ]), where [Di ] are the classes in H 2 (X, Z) dual to submanifolds Di , [X ] ∈ H2 dimC X is the fundamental

348

BORISOV and LIBGOBER

P class of X , and j i j + k = dimC X , are invariants of cobordism of such pairs. (Indeed, if X = ∂Y and j : X → Y , then this number is j ∗ (ci1 ∪ · · · ∪ cik ∪ [E j1 ] ∪ · · · ∪ [E jk ])([X ]) = ci1 ∪ · · · ∪ cik ∪ [D j1 ] ∪ · · · ∪ [D jk ]( j∗ [X ]) = 0 since X is homologous to zero in Y .) The lemma therefore follows if we show that for an almost complex null-cobordant G-manifold X the quotient X/G admits a resolution S ] D), where D = of singularities ( X/G, Di is the exceptional locus such that this pair is cobordant to zero. If X = ∂Y , where Y is a G-manifold, we can construct a resolution of Y/G as follows. Let H be a subgroup of G, and let Y H = {y ∈ Y | Stab y = H }. Then Y H are smooth submanifolds of Y (possibly with boundary) providing a stratification of Y . Let C(H ) be the union of subgroups of G conjugate to H . Then S YC(H ) = H 0 ∈C(H ) Y H 0 is still a submanifold of Y and the group G acts on YC(H ) so that YC(H ) → YC(H ) /G is an unramified cover (of degree [G : H ]). In particular, YC(H ) /G is a smooth manifold and these manifolds for all H ⊂ G provide a stratification of Y/G such that Y/G is equisingular along each stratum YC(H ) /G. A small regular neighborhood of each stratum in Y/G is isomorphic to a bundle ξ H over YC(H ) /G with the fiber isomorphic to V /H , where V is a fiber of the normal bundle to Y H in Y over a point of Y H . (This presentation is independent of a point in Y H , and representations at points of Y H and Y H 0 are isomorphic for conjugate H and H 0 .) For each quotient singularity V /H , let us fix the universal desingularization constructed by Bierstone and Milman (see [8, Theorem 13.2]). Its universality assures that it is equivariant with respect to the centralizer of H in G L(V ). Hence, one can use the transition functions of ξ H to construct the fibration ξ˜ H with the same base as ξ H and having as its fiber the universal resolution of V /H . Moreover, due to universality of canonical resolution (see [8, Theorem 13.2]), this property assures that ξ˜ H corresponding to different classes of conjugate subgroups H can be glued together, ] D), where D yielding an almost complex manifold whose boundary is the pair ( X/G, is the exceptional set of the universal resolution of X/G. This proves the lemma. LEMMA 6.2 The orbifold elliptic genus is an invariant of G-cobordism.

Proof Let X be a null-cobordant G-manifold. Then for each g ∈ G the pair X g , ν(X g , X ), where ν(X g , X ) is the normal bundle of the fixed-point set X g in X , is cobordant to zero as well. Since the contribution of the term in Ellorb corresponding to a conjugacy class [g] is a combination of the products of Chern classes of X g and ν(X g , X ) evaluated on the fundamental class of X g , this contribution is zero. This yields the lemma.

ELLIPTIC GENERA OF SINGULAR VARIETIES

349

Remark 6.3 Unfortunately, the orbifold elliptic genus is not multiplicative. Rather, for Gmanifolds X and Y with an action of G as above, one has Ellorb (X × Y, G × G) = Ellorb (X, G) · Ellorb (Y, G). 6.4 Conjecture 5.1 is true for G = Z/2Z. COROLLARY

Proof This follows from the result of Kosniowski (see [30]) describing generators of Z/ pZcobordisms. If p = 2, then additive generators of the cobordism group in any dimension are toric varieties with the group being a subgroup of the big torus. Hence Proposition 5.3 yields the claim. Acknowledgments. The authors wish to thank Burt Totaro for his helpful comments. We also thank Willem Veys for pointing out an error in the original version of the paper related to the non-log-terminal singularities; see Remark 3.11. References [1]

[2] [3]

[4]

[5]

[6] [7]

[8]

D. ABRAMOVICH, K. KARU, K. MATSUKI, and J. WŁODARSCZYK, Torification and

factorization of birational maps, J. Amer. Math. Soc. 15 (2002), 531 – 572. CMP 1 896 232 321, 327 M. F. ATIYAH and I. M. SINGER, The index of elliptic operators, III, Ann. of Math. (2) 87 (1968), 546 – 604. MR 38:5245 325 V. V. BATYREV, “Stringy Hodge numbers of varieties with Gorenstein canonical singularities” in Integrable Systems and Algebraic Geometry (Kobe/Kyoto, 1997), World Sci., River Edge, N.J., 1998, 1 – 32. MR 2001a:14039 334 , “Birational Calabi-Yau n-folds have equal Betti numbers” in New Trends in Algebraic Geometry (Warwick, England, 1996), London Math. Soc. Lecture Note Ser. 264, Cambridge Univ. Press, Cambridge, 1999, 1 – 11. MR 2000i:14059 333 , Non-Archimedean integrals and stringy Euler numbers of log-terminal pairs, J. Eur. Math. Soc. (JEMS) 1 (1999), 5 – 33. MR 2001j:14018 320, 321, 323, 331, 342 , Canonical abelianization of finite group actions, preprint, arXiv:math.AG/0009043 320 V. V. BATYREV and D. I. DAIS, Strong McKay correspondence, string-theoretic Hodge numbers and mirror symmetry, Topology 35 (1996), 901 – 929. MR 97e:14023 320, 322, 335 E. BIERSTONE and P. MILMAN, Canonical desingularization in characteristic zero by blowing up the maximum strata of a local invariant, Invent. Math. 128 (1997), 207 – 302. MR 98e:14010 348

350

BORISOV and LIBGOBER

[9]

A. BOREL ET AL., Intersection Cohomology (Bern, 1983), Progr. Math. 50, Swiss

[10]

L. A. BORISOV, Vertex algebras and mirror symmetry, Comm. Math. Phys. 215 (2001),

[11]

L. A. BORISOV and P. E. GUNNELLS, Toric varieties and modular forms, preprint,

[12]

L. A. BORISOV and A. LIBGOBER, Elliptic genera of toric varieties and applications to

Seminars, Birkh¨auser, Boston, 1984. MR 88d:32024 321, 325, 334 517 – 557. MR 2002f:17046 323, 335 arXiv:math.NT/9908138 323, 343, 346, 347

[13] [14] [15] [16]

[17] [18] [19]

[20] [21]

[22] [23] [24]

[25]

[26]

mirror symmetry, Invent. Math. 140 (2000), 453 – 485. MR 2001j:58037 319, 323, 333, 338, 344, 345, 346 , McKay correspondence for elliptic genera, preprint, arXiv:math.AG/0206241 323 K. CHANDRASEKHARAN, Elliptic Functions, Grundlehren Math. Wiss. 281, Springer, Berlin, 1985. MR 87e:11058 324, 326, 345 ´ and S. MORI, Higher-Dimensional Complex Geometry, H. CLEMENS, J. KOLLAR, Ast´erisque 166, Soc. Math. France, Montrouge, 1988. MR 90j:14046 325, 329 V. I. DANILOV, The geometry of toric varieties (in Russian), Uspekhi Mat. Nauk 33, no. 2 (1978), 85 – 134; English translation in Russian Math. Surveys 33, no. 2 (1978), 97 – 154. MR 80g:14001 342 J. DENEF and F. LOESER, Germs of arcs on singular algebraic varieties and motivic integration, Invent. Math. 135 (1999), 201 – 232. MR 99k:14002 333 , Motivic integration, quotient singularities and the McKay correspondence, preprint, arXiv:math.AG/9903187 320, 321, 323, 342 R. DIJKGRAAF, G. MOORE, E. VERLINDE, and H. VERLINDE, Elliptic genera of symmetric products and second quantized strings, Comm. Math. Phys. 185 (1997), 197 – 209. MR 98g:81191 319, 335, 338 M. EICHLER and D. ZAGIER, The Theory of Jacobi Forms, Progr. Math. 55, Birkh¨auser, Boston, 1985. MR 86j:11043 324 ¨ G. ELLINGSRUD, L. GOTTSCHE, and M. LEHN, On the cobordism class of the Hilbert scheme of a surface, J. Algebraic Geom. 10 (2001), 81 – 100. MR 2001k:14005 320 J. FOGARTY, Algebraic families on an algebraic surface, Amer. J. Math. 90 (1968), 511 – 521. MR 38:5778 320 W. FULTON, Intersection Theory, 2d ed., Ergeb. Math. Grenzgeb. (3) 2, Springer, Berlin, 1998. MR 99d:14003 327, 328 ¨ L. GOTTSCHE and W. SOERGEL, Perverse sheaves and the cohomology of Hilbert schemes of smooth algebraic surfaces, Math. Ann. 296 (1993), 235 – 245. MR 94i:14026 320 V. GRITSENKO, Elliptic genus of Calabi-Yau manifolds and Jacobi and Siegel modular forms (in Russian), Algebra i Analiz 11, no. 5 (1999), 100 – 125; English translation in St. Petersburg Math. J. 11, no. 5 (2000), 781 – 804. MR 2001i:11051 324 A. GROTHENDIECK, Sur quelques points d’alg`ebre homologique, Tˆohoku Math. J. (2) 9 (1957), 119 – 221. MR 21:1328 325

ELLIPTIC GENERA OF SINGULAR VARIETIES

351

[27]

F. HIRZEBRUCH, Topological Methods in Algebraic Geometry, Classics Math.,

[28]

¨ F. HIRZEBRUCH and T. HOFER , On the Euler number of an orbifold, Math. Ann. 286

[29]

A. KERBER, Representations of the Permutation Groups, I, Lecture Notes in Math.

[30]

C. KOSNIOWSKI, Generators of the Z / p bordism ring: Serendipity, Math. Z. 149

[31]

I. G. MACDONALD, The Poincar´e polynomial of a symmetric product, Proc. Cambridge

[32]

F. MALIKOV, V. SCHECHTMAN, and A. VAINTROB, Chiral de Rham complex, Comm.

[33]

J. MCKAY, “Graphs, singularities, and finite groups” in The Santa Cruz Conference on

Springer, Berlin, 1995. MR 96c:57002 324 (1990), 255 – 260. MR 91g:57038 320, 322 240, Springer, Berlin, 1971. MR 48:4098 341 (1976), 121 – 130. MR 53:11633 349 Philos. Soc. 58 (1962), 563 – 568. MR 26:764 320 Math. Phys. 204 (1999), 439 – 473. MR 2000j:17035a 323

[34] [35] [36] [37] [38]

Finite Groups (Santa Cruz, Calif., 1979), Proc. Sympos. Pure Math. 37, Amer. Math. Soc., Providence, 1980, 183 – 186. MR 82e:20014 320 P. H. SIEGEL, Witt spaces: a geometric cycle theory for K O-homology at odd primes, Amer. J. Math. 105 (1983), 1067 – 1105. MR 85f:57011 334 B. TOTARO, Chern numbers of singular varieties and elliptic homology, Ann. of Math. (2) 151 (2000), 757 – 791. MR 2001g:58037 321, 331, 333 C. T. C. WALL, Cobordism of pairs, Comm. Math. Helv. 35 (1961), 135 – 145. MR 23:A2221 347 W. WANG and J. ZHOU, Orbifold Hodge numbers of the wreath product orbifolds, J. Geom. Phys. 38 (2001), 152 – 169. MR 2002g:32034 341 E. ZASLOW, Topological orbifold models and quantum cohomology rings, Comm. Math. Phys. 156 (1993), 301 – 331. MR 94i:32045 322, 335

Borisov Department of Mathematics, Columbia University, New York, New York 10027, USA; [email protected] Libgober Department of Mathematics, University of Illinois, Chicago, Illinois 60607, USA; [email protected]

ERGODIC PROPERTIES OF LINEAR ACTIONS OF (2 × 2)-MATRICES F. LEDRAPPIER and M. POLLICOTT

Abstract In this paper we consider the ergodic properties of linear actions on the plane of ( 2 × 2)-matrices with entries in the complex numbers, quaternions, or Clifford numbers. 0. Introduction In this paper we study ergodic properties of linear actions of discrete groups of (2×2)matrices with determinant 1. Consider the usual linear action on the real plane R2 of a discrete subgroup 0 in the (2 × 2)-matrices with real entries SL(2, R) given by   a b γ : (x1 , x2 ) 7→ (ax1 + bx2 , cx1 + d x2 ), where γ = ∈ 0. c d The action by such matrices clearly preserves the usual Lebesgue measure on the plane. It follows from work of G. Hedlund [18] that if 0 is a cocompact group, or more generally a lattice, then this linear action is ergodic.∗ This result was extended by F. Ledrappier in [23] to the case of normal subgroups 0 G 0 for which 0/0 is an infinite abelian subgroup. In this note we extend these ergodicity results to discrete subgroups of (2 × 2)matrices over other fields. The simplest case relates to (2 × 2)-matrices with complex entries. In Theorem 1 we show the ergodicity of the linear actions on the complex plane C2 by discrete cocompact groups 0 and by normal subgroups 0 G 0 for which 0/0 is an infinite abelian subgroup. The next most obvious case deals with discrete subgroups of (2×2)-matrices where the entries are quaternions, where the nonabelian nature of the entries provides additional features of interest in the analysis. The corresponding ergodicity result for the linear actions on the quaternionic plane H 2 is described in Theorem 2. Finally, as we shall see, there is a very rich general framework into which many of these results fit, involving the study of matrices whose entries are necessary, we simply replace 0 by h0, −I i. We can then identify 0 ⊂ PSL(2, R) = SL(2, R)/{I, −I } with orientation-preserving isometries of H2 .

∗ If

DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 2, Received 7 September 2001. Revision received 23 January 2002. 2000 Mathematics Subject Classification. Primary 37C85, 37C35; Secondary 37A45, 28D99. 353

354

LEDRAPPIER and POLLICOTT

Figure 1. Part of the orbit of (1, 1) ∈ R2 under the linear action by a triangle group

Clifford numbers. The ergodicity results at this level of generality are described in Theorem 3. It is interesting to note that, although the quaternions are also examples of Clifford numbers, the action derived as a special case of the general construction is different from the usual action described in Theorem 2. Perhaps of interest equal to that of the results we present is their method of proof. In the case of real matrices, there is a useful correspondence between the linear action of SL(2, R) and the action of the same group on the space of horocycles in the Poincar´e half-plane H2 by hyperbolic motions (see [16], [22], [23]). The group 0 is a Fuchsian group corresponding to a compact surface, and 0 corresponds to a Zd -cover. The key idea in considering matrices over different fields is the need to develop a suitable geometric interpretation. In general, the linear action corresponds to an action on an appropriate space of frames on horospheres in higher dimensional Poincar´e halfspaces Hn+1 . Our method also gives ergodicity for many variable curvature examples (see Th. 6). We also consider the distribution of each orbit under such linear actions. For each γ ∈ 0, we can let ||γ || be its norm. Ledrappier considered in [22] the distribution of the orbits γ (x, y) of a point (x, y) ∈ R2 under the linear action of a cocompact group 0 ⊂ SL(2, R), ordered by the norm ||γ || (see Fig. 1). Using a different technique, A. Nogueira studied the case of 0 = SL(2, Z) in [26]. We extend these results to the general case of discrete subgroups of matrices over Clifford numbers (see Ths. 4 and 5).

ERGODIC PROPERTIES OF LINEAR ACTIONS

355

1. Geodesic and frame flows In this section we recall the properties of geodesic flows and frame flows for negatively curved manifolds. We are particularly interested in the ergodic properties of these flows, and we are particularly interested in the case of orientable (n + 1)dimensional manifolds V with constant sectional curvatures, where the stable manifolds have a particularly simple geometric (and algebraic) form. Let us denote by  Hn+1 = (x1 , . . . , xn+1 ) ∈ Rn+1 : xn+1 > 0 the usual (n + 1)-dimensional hyperbolic space, equipped with the Poincar´e metric 2 )/x 2 . The discrete groups 0 and 0 are identified with ds 2 = (d x12 + · · · + d xn+1 n+1 discrete subgroups of orientation-preserving isometries G = Isom(Hn+1 ) of Hn+1 , b = and we can associate to the groups 0 and 0 manifolds V = Hn+1 / 0 and V n+1 H /0. Let St1 (V ) be the unit tangent bundle, and let Stn+1 (V ) be the space of (positively oriented) orthonormal (n + 1)-frames. In particular, each (n + 1)-frame (v1 , . . . , vn+1 ) ∈ St1 (V ) × · · · × St1 (V ) projects to the same point x ∈ V under the canonical projection from St1 (V ) to V . The frames Stn+1 (V ) form a fiber bundle over St1 (V ) with a natural projection π : Stn+1 (V ) → St1 (V ) which selects the first vector in the frame, that is, π(v1 , . . . , vn+1 ) = v1 . The associated structure group acts on each fiber by rotating the frames about the first vector v1 . In particular, we can identify each fiber π −1 (v), v ∈ St1 (V ), with the compact group SO(n). The geodesic flow gt : St1 (V ) → St1 (V ) is defined by parallel transporting v along the geodesic γ : R → V satisfying v = γ˙v (0) for time t. The frame flow f t : Stn+1 (V ) → Stn+1 (V ) acts on frames (v1 , . . . , vn+1 ) ∈ Stn+1 (V ) by parallel transporting for time t the frame along the geodesic γv1 : R → V satisfying v1 = γ˙v1 (0). The geodesic flow gt preserves the (normalized) Liouville measure µ. There is a natural invariant probability measure ν for the frame flow f t , locally described as dν = dµ × d λSO(n) , where λSO(n) denotes the normalized Haar measure on SO(n). The following ergodicity results are known for frame flows (cf. [12]). 1.1 The frame flow f t : Stn+1 (V ) → Stn+1 (V ) is ergodic. PROPOSITION

The strong stable manifold for the geodesic flow is defined by  Wgss (x) = y ∈ St1 (V ) : d(gt x, gt y) → 0, as t → +∞ for x ∈ St1 V. (Here d denotes the Riemannian metric on St1 (V ).) For the frame flow f t : Stn+1 (V ) → Stn+1 (V ) on the compact manifold V , we can define its strong stable

356

LEDRAPPIER and POLLICOTT

manifolds by  W ss for x ∈ Stn+1 (V ). f (x) = y ∈ Stn+1 (V ) : d( f t x, f t y) → 0 as t → +∞ (Here d denotes the Riemannian metric on Stn+1 (V ).) b) → Stn+1 (V b) on V b. There We can similarly define a frame flow b f t : Stn+1 (V ss b are corresponding foliations of Stn+1 (V ) by stable manifolds W b (x). The following f result is also proved in §8. PROPOSITION 1.2 b) and Stn+1 (V ) by their strong stable manifolds are ergodic. The foliations of Stn+1 (V

The following geometric point of view of these foliations is useful. The strong stable manifold Wgss (x) corresponds to the 0-quotient of a horosphere in Hn+1 . We can identify the stable manifolds W ss f (x) with families of frames on these horospheres on V , which are related by some parallel transport along the horosphere. Algebraically, we can describe the geodesic and frame flows on compact (n + 1)dimensional manifolds with constant negative curvatures in terms of discrete 0 < G subgroups of groups of (2 × 2)-matrices G with appropriate entries (e.g., complex numbers, Clifford numbers, etc.) corresponding to isometries on Hn+1 . More precisely, we can identify Hn+1 = G/K and V = 0\G/K , where K < G is a maximal compact subgroup of G. We can then identify Stn+1 (V ) with the cosets 0\G, and then the frame flow corresponds to the algebraic flow  t/2  e 0 0γ 7→ 0γ gt for gt = , where t ∈ R. 0 e−t/2 The stable manifolds for this flow can then be identified with the double cosets   1 ∗ . 0γ N , where N = 0 1 2. The linear action of complex matrices We begin by considering the case of the group of (2 × 2)-matrices of determinant 1 with complex entries. This illustrates the principles of the proof in the more general case, but since the associated compact fiber group SO(2) is abelian for the corresponding 3-manifolds, we are able to take advantage of some simplifying features. Each matrix γ ∈ SL(2, C) corresponds to an isometry of H3 by the conformal action defined on ∂H3 = C ∪ {∞} given by   az + b a b γ : z 7→ , γ = ∈ SL(2, C). c d cz + d

ERGODIC PROPERTIES OF LINEAR ACTIONS

357

We can associate to each γ ∈ SL(2, C) the frame in H3 which is the isometric image of the reference frame based at the reference point (0, 0, 1) ∈ H3 . Algebraically, the parallel transportation of frames along horospheres corresponds to the right action by
matrices of the form 10 1z , z ∈ C. In particular, the double cosets     1 z 0γ :z∈C 0 1

(2.1)

correspond to families of equivalent frames on horospheres on the compact 3dimensional manifold V = 0\H3 . The stable manifolds for the algebraic (frame) flow f t : 0\ SL(2, C) → 0\ SL(2, C) defined by  t/2  e 0 0γ 7→ 0γ gt , where gt = , 0 e−t/2 are precisely of the form (2.1), and, by Proposition 1.2, this equivalence relation is ergodic. We denote C2∗ = C2 − {(0, 0)}. For the linear action γ : C2∗ → C2∗ , the stabilizer of (1, 0) ∈ C2∗ consists of the subgroup of elements γ such that γ (1, 0) = (1, 0), and it takes the form    1 z W = Stab(1, 0) = :z∈C . (2.2) 0 1 LEMMA 2.1 The map C2∗ → G/W , given by

(x, y) 7→



x y

 −y −1 W 0

if y 6= 0

and 

x (x, 0) 7→ 0

0 x −1

 W

otherwise,

is an equivariance between the actions of G = SL(2, C) on C2∗ and G/W . We prove a more general result later. If we quotient out by (x, y) ∼ (−x, −y), the resulting equivariance between ¯ C2∗ / ∼ and G/hW, −I i is better suited for studying the linear actions of 0 and 0. Thus comparing Lemma 2.1 with the ergodicity of the foliation by stable manifolds in Proposition 1.2, and their interpretation as cosets 0γ N , shows the corresponding ergodicity of the linear action.

358

LEDRAPPIER and POLLICOTT

THEOREM 1 Let 0 be a cocompact lattice in SL(2, C) with −I ∈ 0. Let 0 G 0 be a normal subgroup with 0/0 = Zd . The linear actions of both 0 and 0 on C2∗ are ergodic.

Example T. Jørgensen gave a simple geometric construction of examples of cocompact Kleinian groups 0 (which are actually Z-extensions of surface groups). Let us define a real and a complex number by p p √ √ 10 + 2 17 + −6 + 2 17 ρ= 4 and p p √ √ 10 − 2 17 + i 6 + 2 17 p x= . √ 2 −6 + 2 17 We can introduce matrices T, X ∈ SL(2, C) as T =

 ρ 0

0 1 ρ

 and

X=

 (1+i) − √ x 2

1

 −(1 + x 2 ) . (1−i) √ x 2

Jørgensen [19] shows that the group 0 = hT, X i generated by T and X is a cocompact Kleinian group. If we introduce the matrix  Y =

−(1−i) √ x¯ 2 (1+i) √ (ρ −2 − 1)x 2

 √ (ρ 2 − 1)x(1 + x 2 ) − (1−i) 2 , (1+i) √ x¯ 2

then the results of this section apply to the subgroup 0 = hT, Y i generated by T and Y. 3. The linear action of quaternion matrices Let H = {q = x + i y + j z + kw} denote the quaternions with the standard relations i j = k, jk = i, ki = j, and i 2 = j 2 = k 2 = −1. We consider a linear action by matrices   q1 q2 γ = , (3.1) q3 q4 where the entries q1 , . . . , q4 ∈ H . The nonabelian nature of H is such that some additional care is required in defining the determinant. Let us write qi = u i + jvi for i = 1, . . . , 4, where u i , vi ∈ C. Following the usual convention, we can introduce the following definition.

ERGODIC PROPERTIES OF LINEAR ACTIONS

359

Definition The group SL(2, H ) consists of those (2 × 2)-matrices (3.1) for which we have  u1 u 3 det   v1 v3

u2 u4 v2 v4

−v 1 −v 3 u1 u3

 −v 2 −v 4   = 1. u2 u4

Observe that SL(2, H ) has 15 real dimensions. For later use, we state the following simple fact as a lemma. LEMMA 3.1 In the special case when q3 = 0, we have

 u1 0 det   v1 0

u2 u4 v2 v4

−v 1 −0 u1 0

 −v 2 −v 4   = |q1 |2 · |q4 |2 = 1, u2 u4

(3.2)

where we write |qi |2 = u i u i + v i vi = |u i |2 + |vi |2 for i = 1, . . . , 4. The natural linear action SL(2, H ) : H 2 → H 2 is given by   q1 q2 : (x1 , x2 ) 7→ (q1 x1 + q2 x2 , q3 x1 + q4 x2 ). q3 q4 Let us denote    1 0 2 SR = : |q| = 1 . 0 q We can identify γ ∈ SL(2, H ) with conformal maps on R4 = H . The action of S R given by q 0 7→ q 0 q −1 is called a right screw. There is a corresponding action by    q 0 2 SL = : |q| = 1 0 1 on H by q 0 7→ qq 0 , which is called a left screw. Every rotation in SO(4) is conjugate to the composition of two rotations in two orthogonal 2-dimensional planes (with an associated sense). The left screw corresponds to the 3-dimensional group of rotations by the same angle in each of two such planes in the same sense. The right screw corresponds to the 3-dimensional group of rotations by the same angle in each of two such planes in different senses. Furthermore, these actions of SL and S R commute; that is, SL S R = S R SL .

360

LEDRAPPIER and POLLICOTT

Let N be the 4-dimensional real nilpotent group given by    1 q2 N= : q2 ∈ H . 0 1 Let H∗2 = H 2 − {(0, 0)}. For the linear action, the stabilizer of (1, 0) ∈ H∗2 takes the simple form    1 q2 : |q4 |2 = 1 = S R N , (3.3) W = Stab(1, 0) = 0 q4 where the condition on q4 comes from (3.2). The following result is an analogue of Lemma 2.1. LEMMA 3.2 The map H∗2 → G/W , given by

(x, y) 7→



x y

 −y ∗−1 W 0

if y 6= 0

and 

x (x, 0) 7→ 0

0 x −1

 W

otherwise,

is an equivariance between the actions of G = SL(2, H ) on H∗2 and G/W . Proof If a, c 6= 0, then we can represent γ = ( ac db ) in the form  a c

b d

 =

 a c

−c−1 0

  1 a −1 (b + c−1 q) 0 q

for some q with |q| = 1, where the last matrix lies inside W and, by Lemma 3.1, has determinant 1 as a 4×4 complex matrix. We easily check that canonical representative
a −c−1 has determinant |c| · | − c−1 | = 1. c 0 The right action by γ ∈ SL(2, H ) on cosets can be written       x −y −1 ax + by ∗ ax + by ∗ γ W = W = W, y 0 cx + dy ∗ cx + dy 0 provided y 6= 0, and the last term corresponds to the linear action of γ on (x, y), as required. The other case is similar.

ERGODIC PROPERTIES OF LINEAR ACTIONS

361

Again, if we quotient out by (x, y) ∼ (−x, −y), the resulting equivariance between ¯ H∗2 /∼ and G/hW, −I i is better suited for studying the linear actions of 0 and 0. Consider the maximal compact subgroup K given by    q1 q2 2 2 2 2 K = : |q1 | + |q2 | = 1 = |q3 | + |q4 | and q1 q 3 + q2 q 4 = 0 q3 q4 (which is isomorphic to the 10 real dimensional group SO(5)), and consider the 1dimensional real abelian group A given by    q1 0 : q ∈ R A= ∗ . 1 0 q1−1 The centre of K relative to A is the subgroup M ⊂ K defined by    q1 0 2 2 M= : |q1 | = 1 = |q4 | , 0 q4 which is isomorphic to SO(4). Given a discrete subgroup 0 ⊂ SL(2, H ) with −I ∈ 0, we can identify 0\SL(2, H ) with the frame bundle St5 (V ) over the 5-dimensional manifold V = 0\SL(2, H )/K . Similarly, for a normal subgroup 0 G 0, we can b) over the 5-dimensional manifold identify 0\SL(2, H ) with the frame bundle St5 (V b V = 0\SL(2, H )/K . We can identify the nilpotent group N with a translation of frames along horospheres and identify S R with a subgroup of rotations of frames. Since SL and S R commute, SL(2, H )/S R can be identified with an SL -bundle over St1 (H5 ), where SL is isomorphic to the compact group SO(3). We can therefore identify 0\SL(2, H )/S R b), and since S R gt = gt S R , the frame flow b with an SL -bundle over St1 (V f t : 0γ 7→ 0γ gt factors down to f t : 0γ S R 7→ 0γ gt S R , say, which corresponds to the SL extension of the geodesic flow f t : 0γ M 7→ 0γ gt M. The cosets 0γ W correspond to the stable manifolds of the flow f t , and thus they are the quotients of the stable manifolds of the frame flow b f t . In particular, ergodicity of the stable foliations for the flow b f t (by Prop. 1.2) implies the corresponding result for f t . Therefore we have the following result. 2 Let 0 be a cocompact lattice in S L(2, H ) with −I ∈ 0. Let 0 G 0 be a normal subgroup with 0/0 = Zd . The linear actions of 0 and 0 on H 2 are ergodic. THEOREM

4. Linear actions by Clifford matrices The Clifford numbers give a natural general framework into which we can fit both the complex numbers C and the quaternions H . For n ≥ 1, consider n − 1 elements

362

LEDRAPPIER and POLLICOTT

i 1 , . . . , i n−1 which satisfy the relations ir i s = −i s ir and ir2 = −1. The Clifford numbers V n are the n-dimensional real vector space a0 + a1 i 1 + · · · + an−1 i n−1 with a0 , . . . , an−1 ∈ R. The associated Clifford algebra∗ Cn is the 2n -dimensional space consisting of elements of the form X a = a0 + aiv1 ···iv p i v1 · · · i v p . v1 <···
It is easy to see that C1 can be identified with R and that C2 can be identified with C. We can identify C3 with H , where i, j, k are represented by i 1 , i 2 , i 1 i 2 . There is a simple conjugation a 7→ a ∗ which reverses the order of the factors in each term i v1 · · · i v p to give i v p · · · i v1 . Let 0n be the set of products of nonzero Clifford numbers. A useful observation is that x ∈ V n if and only if axa ∗ ∈ V n for a ∈ 0n (see [2, §1.5]). Definition We define the Clifford matrices SL(2, 0n ) to consist of (2 × 2)-matrices   a b γ = , a, b, c, d ∈ 0n ∪ {0}, c d such that ad ∗ − bc∗ = 1 and ab∗ , cd ∗ ∈ V n . The isometries of Hn+1 correspond naturally to conformal maps on the n-dimensional boundary ∂Hn+1 = Rn ∪ {∞}. Moreover, by [2, Th. B], the group PSL(2, 0n ) is isomorphic to this M¨obius group of conformal maps on Rn ∪ {∞}. We can naturally identify (a0 , · · · , an−1 ) with a0 + a1 i 1 + · · · + an−1 i n−1 . In particular, the matrices
1β , where β ∈ V n , give translation by β, and they correspond to parallel transla0 1 tion on horospheres (see [2]). Given a cocompact discrete subgroup 0 ⊂ SL(2, 0n ), the double cosets 0γ W correspond to families of equivalent frames on horospheres on the compact (n + 1)-dimensional manifold V = 0\Hn+1 (see [2]). We can consider the algebraic (frame) flow f t : 0\ SL(2, 0n ) → 0\ SL(2, 0n ) defined by  t/2  e 0 0γ 7→ 0γ gt , where gt = . 0 e−t/2 The stable manifolds for this flow are double cosets 0\ SL(2, 0n )/W , where    1 β W = : β ∈ Vn . 0 1 ∗ For

[24].

an account of the development of this theory, we refer the reader to the correspondence of “R. Lipschitz”

ERGODIC PROPERTIES OF LINEAR ACTIONS

363

To study the linear action, we need to consider the restriction to a subspace  An = (x, y) ∈ (0n ∪ {0}) × (0n ∪ {0}) : ∃e ∈ V n with y ∗ x = e and to define the (restricted) linear action 0 × An → An in the obvious way by   a b : (x, y) 7→ (ax + by, cx + dy). c d To see that this is well defined, observe that for y 6= 0 we have y ∗ x ∈ V n if and only if x y −1 ∈ V n . If c(x y −1 ) + d 6= 0, then (a(x y −1 ) + b)(c(x y −1 ) + d)−1 ∈ V n by [2, §2.4]. The other cases are simpler. If we define Q : (0n ∪ {0}) × (0n ∪ {0}) → 0n by Q(x, y) = y ∗ x − x ∗ y, then we have the equivalent definition  An = (x, y) ∈ (0n ∪ {0}) × (0n ∪ {0}) : Q(x, y) = 0 . Clearly, if y ∗ x ∈ V n , then we have Q(x, y) = 0. The reverse implication follows from an application of a theorem of Lipschitz and T. Vahlen (see [4, §3.2]), which gives that a = y ∗ x ∈ 0n is uniquely determined by the coefficients a0 , a j , and a jk . More precisely, since (y ∗ x)∗ = x ∗ y = y ∗ x, we see that ai j = 0, and so y ∗ x = P a0 + nj=1 a j i j ∈ V n . Examples Notice that A1 = R2 and A2 = C2 , but A3 ⊂ H 2 . However, observe that SL(2, 03 ) is 10-dimensional; thus it is different from SL(2, H ), as defined in §3. We can denote An ∗ = An − {(0, 0)}. 4.1 The map An ∗ → G/W , given by LEMMA



x (x, y) 7→ y

 −y ∗−1 W 0

if y 6= 0

and (x, 0) 7→



x 0

0 x −1

 W

otherwise,

is an equivariance between the action of SL(2, 0n ) on An ∗ and G/W .

364

LEDRAPPIER and POLLICOTT

Proof Assume that a, c = 6 0. Since cd ∗ ∈ V and ad ∗ − bc∗ = 1, we can use the decomposition      a b a −c∗−1 1 a −1 (b + c∗−1 ) = c 0 0 1 c d to see that every coset in SL(2, 0n )/W has a canonical coset representative of the ∗−1
form ac −c0 W , where c∗ a ∈ V n . Moreover, if we consider the right action by γ on cosets, we can write       ax + by ∗ x −y ∗−1 ax + by ∗ γ W = W = W, y 0 cx + dy ∗ cx + dy 0 provided y 6= 0, and the last term corresponds to the linear action of γ on (x, y), as required. Similarly, if c = 0 and a 6= 0, then since ab∗ ∈ V we have a −1 b ∈ V and we can write      a b a 0 1 a −1 b = , 0 a ∗−1 0 a ∗−1 0 1 and if a = 0 and c 6= 0, then since cd ∗ ∈ V we have c−1 d ∈ V and we can write      0 −c∗−1 0 −c∗−1 1 c−1 d = . c d c 0 0 1 The equivariance is easily checked in these cases too. If we quotient out by (x, y) ∼ (−x, −y), the resulting equivariance between An ∗ /∼ ¯ Under this and G/hW, −I i is better suited for studying the linear actions of 0 and 0. above equivariance map, the Haar measure induces a measure µ on An /∼ which is absolutely continuous. The following result is a natural generalization of Theorem 1 and is again a consequence of Proposition 1.2. THEOREM 3 Let 0 be a cocompact lattice in SL(2, 0n ) with −I ∈ 0. Let 0 G 0 be a normal subgroup with 0/0 = Zd . The linear actions of 0 and 0 on An are ergodic.

5. The distribution of orbits The distribution of the orbits 0(x, y) can most easily be described by a geometric interpretation of the actions. At the level of generality of Clifford matrices, the proof is essentially a straightforward generalization of the argument in [22] for SL(2, R). The results for complex and quaternionic matrices follow as corollaries.
0 The action on V n of the matrices α0 α ∗−1 with |α| = 1 correspond to elements n+1 M ⊂ G, which rotate the reference frame in H by an element of SO(n). Using the

ERGODIC PROPERTIES OF LINEAR ACTIONS

365

K AN decomposition, we can write each γ ∈ SL(2, 0n ) as a product of matrices     a b λ 0 1 β , (5.1) γ = c d 0 1/λ 0 1 where λ = ed/2 (for d ∈ R+ ), β ∈ Vn , and the first matrix lies in the maximal compact subgroup K . These transformations on V n ∪ {∞} can be interpreted in terms of the reference frame in Hn+1 , on the reference horosphere based at ∞. The transformations are, respectively, rotation by an element of SO(n + 1), transportation under the geodesic flow by gd , and translation on the horosphere by β ∈ V n = Rn . If γ (1, 0) = (x, y), then we can associate to (x, y) ∈ An the image of the reference horosphere, which is the unique horosphere (1) based at γ (∞) = x y −1 ∈ V n ∪ {∞} (see [2, §2.2]), (2) at a (signed) hyperbolic distance d = log(|x|2 + |y|2 ), called the Busemann function. Furthermore, we can associate those frames on this horosphere which are parallel transports of (3) the reference frame rotated by x 7→ αxα ∗ , where α ∈ 0n , |α| = 1 (see [2, §2.6]). Let X = (x, y) ∈ An ∗ . We are interested in the distribution of orbits under 0. We have the following property. THEOREM 4 Let 0 be a cocompact lattice in SL(2, 0n ), and let f be a continuous function with compact support on An ∗ . Then there is a constant C > 0 such that Z X C f (Y ) 1 f (γ X ) = dλ(Y ), lim n n T →∞ T |X | An |Y |n γ ∈0:||γ ||≤T

where |X | =

p

x 2 + y 2 and λ is volume on An .∗

Remark. For n = 1, this is the result of [22].† For n = 2, we obtain the corresponding result for SL(2, C). The constant C can be explicitly computed to be C = 2ωn /Vol(0\ SL(2, 0n )), where ωn is the volume of the unit ball in Rn . Proof To prove Theorem 4, we can define a map 9 : An ∗ → SL(2, 0n ), where we asso
0 ciate to a point X = (x, y) = γ (1, 0) ∈ An ∗ the product 9(X ) = ( ac db ) λ0 1/λ ∈ ∗ One can compare this with estimates on π(T ) = #{g ∈ 0 : ||g|| ≤ T }. There exists C > 0 such that π(T ) ∼ C T n+1 (see [32]). † The constant given in [22, Rem. 1] is unfortunately off by a factor 2 (cf. [26]).

366

LEDRAPPIER and POLLICOTT

SL(2, 0n ). Geometrically, 9(X ) is the reference frame rotated by an element of K and moved forward by the geodesic flow gd , where λ = ed/2 . The action of an element γ ∈ SL(2, 0n ) on Hn+1 carries the horosphere supporting the frame 9(X ) to another horosphere containing the frame γ 9(X ). Moreover, the frames are related by a horospherical translation γ 9(x)h s = 9(γ x) for some s = s(γ , x). We have the following estimate. LEMMA 5.1
0 0 With the above notation, let ( ac db ) λ0 1/λ represent X ∈ An ∗ , let ac0 represent γ X , and let s ∈ V n be such that     0    a b λ 0 a b 0 λ0 0 1 s γ = 0 ; c d 0 1/λ c d0 0 1/λ0 0 1

b0 d0




λ0 0 0 1/λ0




then ||γ ||2 = |X |2 |γ X |2 |s|2 + |X |2 /|γ X |2 + |γ X |2 /|X |2 . Proof We have  0 a c0

b0 d0

−1  a γ c

b d



 0   −1 λ 0 1 s λ 0 0 1/λ0 0 1 0 1/λ  0  λ /λ λλ0 s . = 0 λ/λ0 =

The equality follows by taking the norm of each side. Following [22, p. 63], we can write that for any small region D ⊂ An ∗ we have ||γ || ≤ T and γ X ∈ D if and only if 9(X )h s ∈ γ 9(D) for some s with |s| ≤ T /(|X ||γ X |). This allows us to replace Z X f (γ X ) by f˜(0 X h s ) ds, (5.2) γ ∈0:||γ ||≤T

|s|≤T /(|X ||Y |)

where h s is the right multiplication by h s = ( 10 1s ) ∈ W and where f˜ is a suitable function with small support on 0\ SL(2, 0n ). We also require the following result, which follows easily from [14]. 5.2 Let 0 be a cocompact lattice, and let µ denote the Haar measure on 0\ SL(2, 0n ). For any f¯ ∈ C 0 (0\ SL(2, 0n )), we have Z Z 1 ¯(0gh s ) ds → lim f f¯ dµ, T →+∞ ωn T n |s|≤T LEMMA

ERGODIC PROPERTIES OF LINEAR ACTIONS

367

where ωn is the volume of the unit ball in Rn , and the convergence is uniform for any g ∈ SL(2, 0n ). In particular, if Z ∈ 0\ SL(2, 0n ) is a coset, then by Lemma 5.2, for a continuous function f¯ on 0\ SL(2, 0n ), we have Z Z 1 ¯ f (Z h s ) ds = f¯ dµ. (5.3) lim T →∞ wn T n |s|≤T Then, since the coset Z W corresponds to a point in An by comparing (5.2) and (5.3), the conclusion of Theorem 4 holds. 6. Symbolic dynamics The proof of Proposition 1.2 makes essential use of the symbolic models of the associated flows. The geodesic flow gt : St1 (V ) → St1 (V ) has a particularly simple symbolic model, which we now describe. Let A be a k × k aperiodic matrix with entries 0 and 1, and define the space ∞ n o Y X A = x = (xn )∞ ∈ {1, . . . , k} : A(x , x ) = 1, n ∈ Z n n+1 n=−∞ n=−∞

P |n| with metric d(x, y) = ∞ n=−∞ (1 − δ(x n , yn ))/2 , where δ(i, j) is the Dirac delta function on the symbol space. Let σ : X A → X A be the two-sided subshift of finite type defined by (σ x)n = xn+1 . Given a H¨older continuous function r : X A → R, we define X rA = {(x, u) ∈ X A × R : 0 ≤ u ≤ r (x)}, where we identify (x, r (x)) = (σ x, 0). We can associate the suspended flow σtr : X rA → X rA , where σtr (x, u) ( (σ n x, u + t − r n (x)) = (σ −n x, u + t + r n (σ −n x))

if r n (x) ≤ u + t ≤ r n+1 (x), if − r n (σ −n x) ≤ u + t ≤ −r n−1 (σ −(n−1) x),

where we denote r n (x) = r (x)+r (σ x)+· · ·+r (σ n−1 x). Let h(σ r ) be the topological entropy of the flow σtr . We let ν be the unique σ -invariant probability measure that satisfies Z Z h(σ, ν) − h r dν ≥ h(σ, m) − h r dm, where m is an arbitrary σ -invariant probability measure with entropy h(σ, m) and h = h(σ r ). We then define a σ r -invariant probability measure on X rA by d µ¯ = R dν × dt/ r dν.

368

LEDRAPPIER and POLLICOTT

PROPOSITION 6.1 (see R. Bowen [7], M. Ratner [29], Bowen and D. Ruelle [9]) Given a geodesic flow gt : St1 (V ) → St1 (V ), there exists a suspended flow σtr : X rA → X rA and a continuous surjective map π : X rA → St1 (V ) such that (i) π is a semiconjugacy (i.e., π ◦ σtr = gt ◦ π), (ii) π : (X rA , µ) ¯ → (St1 (V ), µ) is a isomorphism, and (iii) h(g) = h(σ r ).

Although we need not be concerned with the details of the proof of Proposition 6.1, it is useful for the sequel to have some broad understanding. The sets Ti = π([i] × {0}), i = 1, . . . , k, are local codimension one sections transverse to the geodesic flow gt : M → M. Thus the map σ : X A → X A models the Poincar´e map between these sections, and the transition time for the flow between points π(x) ∈ Tx0 and π(σ x) ∈ Tx1 is given by r (x). The sections can be chosen arbitrarily small, and in later proofs we assume that they are as small as we require. To extend this model to the frame flow f t : Stn+1 (V ) → Stn+1 (V ), we can study a compact group extension of the symbolic flow σ¯ tr : X rA × SO(n) → X rA × SO(n), where σ¯ tr (x, u, α) = (x, u + t, α), subject to the identification (x, r (x), α) ∼ (σ x, 0, 20 (x)α), where 20 : X A → SO(n) is a H¨older continuous function that records the rotation in the frame between b) → Stn+1 (V b) on the Zd -cover can each section. Finally, the frame flow b f t : Stn+1 (V be modeled by a symbolic flow b σtr : X rA × SO(n) × Zd → X rA × SO(n) × Zd , where b σtr (x, u, α, z) = (x, u + t, α, z), subject to the identification (x, r (x), α, z) ∼ (σ x, 0, 20 (x)α, z + f (x)), where f : X A → Zd is a H¨older continuous function that records the Frobenius element in Zd between lifts of each section to the Zd -cover. Let us introduce the space X+ A

n

= x∈

∞ Y n=0

o {1, . . . , k} : A(xn , xn+1 ) = 1, n ∈ Z

ERGODIC PROPERTIES OF LINEAR ACTIONS

369

P∞ |n| with metric d(x, y) = n=0 (1 − δ(x n , yn ))/2 . We can assume, without loss of + generality, that r : X A → R, for example, r (x) = r (y) if xi = yi for i ≥ 0. Since the function f : X A → Zd is locally constant, we can assume by recoding, if necessary, that f (x) = f (x0 , x1 ). The following result appears as [28, App. II]. 6.2 There exists a H¨older continuous function U : X A → SO(n) such that 2(x) = U (σ x)−1 20 (x)U (x) is a H¨older continuous function that depends on future coordinates; that is, we can identify this function with 2 : X + A → SO(n). LEMMA

+ Let σ : X + A → X A be the one-sided subshift of finite type defined by (σ x)n = x n+1 . + This is a local homeomorphism. Writing r : X + A → R and 2 : X A → SO(n), we can define  3 = (x, u, α) ∈ X + A × R × SO(n) : 0 ≤ u ≤ r (x) ,

where one identifies (x, r (x), α) = (σ x, 0, 2(x)α), and  ¯ = (x, u, α, z) ∈ X + × R × SO(n) × Zd : 0 ≤ u ≤ r (x) , 3 A where one identifies (x, r (x), α, z) = (σ x, 0, 2(x)α, z + f (z)). The measure ν on X A induces a unique probability measure on X + A , which, in the interests of reducing notation, we again denote by ν. Remark. We can assume that ν is an equilibrium state for a potential − log u; then we can assume, without loss of generality, that u is normalized; that is, we can assume P that σ y=x u(y) = 1 (see [28]). We define natural measures  dt  dν × R × dλSO(n) r dν

 dt  dν × R × dλSO(n) × dz r dν

and

¯ respectively, where dz is the counting measure on Zd and λSO(n) denotes on 3 and 3, Haar measure on K = SO(n). These correspond to the natural measures on Stn+1 (V ) and Stn+1 (Vˆ ), respectively. We define an equivalence relation on 3 by (x, u, α) ∼ (x 0 , u 0 , α 0 ) if there exist 0 n, n ≥ 0 such that  n n0 0   σ x = σ x , 0

r n (x) − u = r n (x 0 ) − u 0 ,   2n (x)α = 2n 0 (x 0 )α 0 ,

where 2n (x) = 2(σ n−1 x) · · · 2(σ x)2(x). We can apply Proposition 1.1 to deduce the following result.

370

LEDRAPPIER and POLLICOTT

PROPOSITION 6.3 The equivalence relation on 3 is ergodic.

7. Aperiodicity and cross-ratios ¯ by (x, u, α, z) ∼ (x 0 , u 0 , α 0 , z 0 ) if there exist We define an equivalence relation on 3 0 n, n ≥ 0 such that  0 σ n x = σ n x 0 ,    r n (x) − u = r n 0 (x 0 ) − u 0 , 0  2n (x)α = 2n (x 0 )α 0 ,    0  z + f n (x) = z 0 + f n (x 0 ),

where f n (x) = f (x) + f (σ x) + · · · + f (σ n−1 x). The following is a variant of a definition that appears in a paper of Y. Guivarc’h [17]. Definition We say that the equivalence relation is weakly aperiodic if there are only trivial solutions to
h(σ x) = e−ihθ, f (x)i e−itr (x) eiξ R 2(x) h(x), (7.1) \ × Td × R, SO(n) \ where h : X → Ck is a continuous function, (R, θ, t) ∈ SO(n) represents the dual space of the unitary representations for SO(n), and ξ ∈ R. PROPOSITION 7.1 ¯ is weakly aperiodic. The equivalence relation on 3

Proof We want to show that if (R, θ, t, ξ ) satisfies (7.1), then R is trivial. Condition (7.1) then reduces to a one-dimensional identity related only to the underlying geodesic flow. The triviality of the identity in this simpler case follows from the results in [31]. We first fix a choice of Markov section, T = π −1 ([i] × {0}), say, where [i] = {x ∈ X A : x0 = i}, and a dense G δ subset O ⊂ T such that the projection π is oneto-one from π −1 O ⊂ X A to O . We also fix a choice of lifts T˜ and O˜ to St1 Hn+1 and write G for the set of geodesics (x, u) of Hn+1 which intersect O˜ . To fix coordinates on O˜ , let v˜ be the lift of a vector v which lies in O . We can identify the local strong ss (v) stable manifold Wloc ˜ with a neighborhood of γv˜ (−∞) in ∂Hn+1 by the map w 7→ γw (−∞). This gives an orthonormal frame on Tv˜ W ss . Our reference frame is the projection of this frame to T Hn+1 . Given the identification of ∂Hn+1 with Rn ∪ {∞} and any periodic point x ∈ π −1 O with period m, corresponding to a closed geodesic that has a lift γ passing

ERGODIC PROPERTIES OF LINEAR ACTIONS

371

through O˜ with endpoints γ− and γ+ in Rn ∪ {∞}, we can write r m (x) = |γ 0 (γ− )| = el(γ ) , 0 2m 0 (x) = ργ (γ− ),

where ργ (x) denotes the rotation part of the similarity γ 0 (x) (cf. [2, §2.7], where this is reduced to the case when the axis of γ is (0, ∞)). Let U : X A → SO(n) be as in Lemma 6.2, and set h 0 (x) = U (x)h(x) for x ∈ X A . Substituting into (7.1), we see that h 0 satisfies
h 0 (σ x) = e−ihθ, f (x)i e−itr (x) eiξ R 20 (x) h 0 (x). (7.10 ) Let x ∈ π −1 O be a periodic point such that σ m x = x and f m (x) = 0. We can apply (7.10 ) to the points {σ j x, j = 0, 1, . . . , m − 1} to obtain
itr m (x) −imξ R 2m e h 0 (x). 0 (x) h 0 (x) = e If x corresponds to a geodesic γ which satisfies [γ ] = 0 with endpoints (γ− , γ+ ), we write h(γ− , γ+ ) = h(x). We now have |γ 0 (γ− )| = el(γ ) ,
R ργ (γ− ) h(γ− , γ+ ) = eitl(γ ) e−imξ h(γ− , γ+ ).

(7.2)

The following two quantities were introduced by L. Ahlfors [3, (8) and (20)] for four distinct points x, y, u, v ∈ V n . Definition We define the absolute cross-ratio by |x, y, u, v| =

|x − u| · |y − v| ∈ R+ , |y − u| · |x − v|

and we define the angular cross-ratio to be the rotation in SO(n) given by



g(x, y, u, v) = I − 2Q(x − u) I − 2Q(x − v) I − 2Q(y − v) I − 2Q(y − u) , where x 7→ (I − 2Q(a)) denotes the reflection in Rn through the hyperplane containing zero and a. Observe that |x, y, u, v| and g(x, y, u, v) are continuous in their arguments. Following [3, (7) and (19)], we write for any γ M¨obius transformation of Rn ∪ {∞} and x, y ∈ Rn , |γ x − γ y| = |γ 0 (x)|1/2 · |γ 0 (y)|1/2 · |x − y|,
−1
1 − 2Q(γ x − γ y) = ργ (x) 1 − 2Q(x − y) ργ (y).

(7.3)

372

LEDRAPPIER and POLLICOTT

Consider a hyperbolic element A (resp., B) with associated axis (a− , a+ ) (resp., (b− , b+ )). For p large, let x p denote the repelling fixed point of A p B p , and let z p be the corresponding attracting fixed point. Then y p = B p x p is the repelling fixed point for B p A p . Observe that x p = A p y p and that x p → b− , z p → a+ , and y p → a− , as p → ∞. By the chain rule, (A p B p )0 (x p ) = (A p )0 (y p )(B p )0 (x p ). Using (7.3), we obtain  |a − y ||b − x | 2 p p p p + p + p = el(A )+l((B )−l(A B ) , |a+ − x p ||b+ − y p |

ρ(A p B p ) (x p ) = 1 − 2Q(a+ − y p ) ρ A p (a+ ) 1 − 2Q(a+ − x p )

× 1 − 2Q(b+ − x p ) ρ B p (b+ ) 1 − 2Q(b+ − y p ) . (7.4) Assume that the periodic orbits associated to the two hyperbolic elements A, B are such that they both intersect a common interior of a Markov section; then we claim that m(A p B p ) = m(A p )+m(B p ). Without loss of generality, we can choose A and B such that the associated closed geodesics correspond to closed orbits for the geodesic flow gt which pass through the interior of the same Markov section, Ti , say. Assume that the corresponding periodic orbits in X A are repetitions of words x0 · · · xm(A)−1 and y0 · · · ym(B)−1 , where we can assume that x0 = y0 = i. The periodic orbit for gt associated to A p B p is coded by repetitions of the word x0 · · · xm(A)−1 . . . x0 · · · xm(A)−1 y0 · · · ym(B)−1 . . . y0 · · · ym(B)−1 , | {z }| {z } ×p

×p

and, in particular, m(A p B p ) = m(A p ) + m(B p ). To see this, it suffices to check that the associated closed geodesic for this new word lies in the free homotopy class of A p B p ∈ π1 (V ). Following a construction of J. Franks [15], one fixes a reference point p0 ∈ V and chooses paths γi from p0 to the projection of the section Ti into V for 1 ≤ i ≤ k. Whenever A(i, j) = 1, we can define a closed path γi j based at p0 by concatenating γi , a geodesic arc between the projections of Ti and T j , and γ j−1 . (Provided the sections are chosen sufficiently small, we can assume that the homotopy class [γi j ] ∈ π1 (V ) is well defined.) The homotopy class associated to the word x0 · · · xm(A)−1 , for example, is simply the product [γx0 x1 ] · · · [γxm(A)−1 x0 ] ∈ π1 (V ). A second simplification comes from the fact that we can choose a subsequence pi → ∞ such that e−it pi l(A) eimpi ξ ρ A pi (a+ ) → Id, e−it pi l(B) eimpi ξ ρ B pi (b+ ) → Id, as i → ∞.

ERGODIC PROPERTIES OF LINEAR ACTIONS

373

Collecting together these observations, we obtain, for two hyperbolic elements ¯ corresponding to geodesics with endpoints (a− , a+ ) and (b− , b+ ), which A, B of 0, pass through O , |a+ , b+ , a− , b− |2 = lim el(A p→∞

p )+l(B p )−l(A p B p )

as p → ∞

(cf. [20]) and
R g(a+ , b+ , a− , b− ) h(b− , a+ ) = |a+ , b+ , a− , b− |−2it h(b− , a+ ).

(7.5)

By continuity, (7.5) still holds for any limit point of such quadruples (a+ , b+ , a− , b− ). In particular, for any (x, y, u, v) such that the geodesics with endpoint (x, u) and (y, v) pass through O , the vector h(x, v) satisfies
R g(x, y, u, v) h(x, v) = |x, y, u, v|−2it h(x, v). (7.6) To complete the proof, we can fix x and v. By conjugating x to zero and v to ∞, we have a representation R such that there exists a vector h (i.e., the conjugate of h(x, v)) with the property that for any (y, u) in a neighborhood of (0, ∞) we have
R g(0, y, u, ∞) h = |0, y, u, ∞|−2it h. (7.7) The elements g(0, y, u, ∞), where (y, u) satisfy the above restrictions, generate SO(n) (cf. [3, §12]), from which we deduce that either the representation R is one-dimensional or it is trivial. However, the existence of a one-dimensional representation would contradict the simplicity of SO(n) for n ≥ 5 and n = 3. For n = 4, we can write SO(4) = SO(3) × SO(3), and so again there can be no one-dimensional representation. Thus, for n ≥ 3, one deduces that the representation R is trivial. Finally, in the case where n = 2, we see directly from the fact that (7.7) holds on some neighborhood that R is trivial because we have the freedom to change (y, u) so that R(g(0, y, u, ∞)) remains the same but |0, y, u, ∞| changes, or vice versa (cf. [3, (24) and §12]). 8. Exactness ¯ → 3, ¯ t ≥ 0, defined by σ¯ tr (x, s, α, z) = We can consider the semiflow σ¯ tr : 3 (x, s+t, α, z), subject to the identification (x, r (x), α, z) ∼ (σ x, 0, 2(x)α, z+ f (x)). This provides a symbolic model for b f t . We say that σ¯ tr is exact if the tail sigma algebra is trivial. We want to use the weak aperiodicity to establish the following.

374

LEDRAPPIER and POLLICOTT

PROPOSITION 8.1 The flow σ¯ tr is exact.

b) → Moreover, we have that the horospherical foliation for the frame flow b f t : Stn (V b Stn (V ) is ergodic (i.e., Prop. 1.2 is proved) since the tail field of b σt coincides (mod 0) with the σ -algebra of measurable sets that are unions of strong stable manifolds. Proof The proof of Proposition 8.1 depends on a number of lemmas, but it follows the general scheme in [17]. Let u : X + → R be the function dνσ/dν = u, that is, the PA function for which we have the σ y=x u(y) = 1. We denote by P : L 1 (X + A ×R× + d 1 d SO(n) × Z ) → L (X A × R × SO(n) × Z ) the linear operator X
P f (x, u, g, k) = u(y) f y, u − r (y), 2−1 (y)g, k − f (y) . σ y=x

The next lemma is essentially contained in [17] (see also [1]) and is used to prove Proposition 8.1. 8.2 R d Assume that |P n F| dm → 0 as n → +∞ for any F ∈ L 1 (X + A × R × SO(n) × Z ) R satisfying F(x, t, g, z) dλR×G×Zd (t, g, z) = 0 for a.e. x, where λ corresponds to the Haar measure on G, the counting measure on Zd , and the Lebesgue measure on R. Then σ¯ tr is exact. LEMMA

We know that for every point we have | f n (x)| ≤ K 1 n and |r n (x)| ≤ K 2 n, for some constants K 1 , K 2 > 0. For the purposes of the proof, it is better to study the L 2 -norm. The next result relates the L 1 -norm and L 2 -norm. LEMMA 8.3 1 1 1 d Let A ∈ L 1 (X + A ), B ∈ L (R), C ∈ L (G), and D ∈ L (Z ); then there exists K > 0 such that

||P n (A ⊗ B ⊗ C ⊗ D)||1 ≤ K n (d+1) ||P n (A ⊗ B ⊗ C ⊗ D)||2 + o(1). 2

(8.1)

Proof The proof of this lemma follows the lines of that in papers [17] and [1]. In particular,

ERGODIC PROPERTIES OF LINEAR ACTIONS

375

we can bound ||P n (A ⊗ B ⊗ C ⊗ D)||1 Z ≤ I (t, g, z) dλR×G×Zd (t, g, z) {t :|t|≤2K 2 n}×G×{z :|z|≤2K 1 n} Z + I (t, g, z) dλR×G×Zd (t, g, z) R×G×{z :|z|>2K 1 n} Z + I (t, g, z) dλR×G×Zd (t, g, z), {t :|t|>2K 2 n}×G×Zd

(8.2)

where Z


 P n A(·) ⊗ B(t − r n (·) ⊗ C 2−n (·)g ⊗ D z − f n (·) dν . I (t, g, z) = Using the Cauchy-Schwarz inequality, we can bound the first term in (8.2) by p 4K n Card(B(0, 2K 1 n) ∩ Zd ) ||P n (A ⊗ B ⊗ C ⊗ D)||2 ≤ K 0 n d+1 ||P n (A ⊗ B ⊗ C ⊗ D)||2 . For the second term, we use the fact that | f n | ≤ K 1 n to write Z I (t, g, z) dλR×G×Z(t,g,z) R×G×{z : |z|>2K 1 n}  X  ≤ ||A||1 ||B||1 ||C||1 |D(z)| → 0 as n → +∞. {z :|z|>K 1 n} The third term in (8.2) goes to zero as n → +∞ for the same reason. This completes the proof of Lemma 8.3. d We can approximate any F ∈ L 2 (X + A × R × G × Z ) satisfying Z F(x, t, g, z) dλR×G×Zd (t, g, z) = 0

PN for a.e. x, by functions k=0 Ak ⊗ Bk ⊗ Ck ⊗ Dk , where Ak ∈ C α (X + A ), Bk ∈ 1 2 1 L (R) ∩ L (R), Ck ∈ L (G) ∩ L 2 (G), and Dk ∈ L 1 (Zd ) ∩ L 2 (Zd ), such that N

X

Ak ⊗ Bk ⊗ Ck ⊗ Dk < ε.

F − k=0

1

We recall that there are only countably many irreducible unitary representations of G (for any compact Lie group; see [25]). For each 0 ≤ k ≤ N , we can assume that the function Fk = Bk ⊗ Ck ⊗ Dk satisfies the following:

376

LEDRAPPIER and POLLICOTT

bk is compactly supported in R × G b × Td ; in particular, the Fourier transform F there are only finitely many irreducible representations with nonzero coefficient; and bk is disjoint from I = (0 × trivial × 0). (2) the support of F 2 Using the L -norm instead of the L 1 -norm, one can apply the Plancherel formula for the Fourier transform on fibers. More precisely, let us consider representations b × Td , where Rχ : G → U (Eχ ) is a unitary γ = (s, Rχ , w) ∈ R ×\ b G × Zd = R × G representation. For reference, we state the result as a lemma. (1)

8.4 For a compact group G, there are countably many irreducible unitary representations Rχ : G → U (Eχ ). Given C ∈ L 2 (G), we have a unique representation R bχ , χ ∈ G}, b where each C bχ is given by C bχ = C(g)Rχ (g) dλG (g). Then C = {C the Plancherel equality takes the form Z X bχ C bχ∗ ). |C(g)|2 dλG (g) = trace(C LEMMA

G

χ

Definition We associate to a representation b γ the operator Pγˆ : C α (X, C N ) → C α (X, C N ) defined by X
Pγˆ h(x) = u(y)eisr (y) e−ihw, f (y)i Rχ 2−1 (y) h(y). σ y=x

The following lemma relates Pb γ to P. LEMMA 8.5 We can write Z X G×R

=

 |P m (A ⊗ B ⊗ C ⊗ D)(x, t, g, n)|2 dλG (g) dt

n∈Zd

XZ b χ∈G

Td ×R

2 b |b B(s)|2 | D(w)|

nχ X

Pγm ˆ [A(·)C χ ei ], e j


2

dλTd (w) ds.

i, j=1

Proof We first consider the Fourier transform of the function P m (A ⊗ B ⊗ C ⊗ D)(x, t, g, n) X


= u m (y)A(y)B t + r m (y) C 2−m (y)g D n − f m (y) . σ m y=x

ERGODIC PROPERTIES OF LINEAR ACTIONS

377

m b The Fourier transform of D(n − f m (·)) is of the form e−ihw, f (·)i D(w), and the m m isr (·) b Fourier transform of B(t + r (·)) is of the form e B(s). The Fourier coefficients bχ . of C(2−m (·)g) are of the form Rχ (2−m (·))C m The Fourier transform of P (A ⊗ B ⊗ C ⊗ D) in (t, g, n) is then given, for each (s, Rχ , w), by the linear map in Eχ , X
isr m (y) −ihw, f m (y)i b bχ v v 7→ u m (y)A(y) b B(s) D(w)e e Rχ 2−m (y) C

σ m y=x


m b b =b B(s) D(w)P γˆ A(·)C χ v . Finally, we apply the Plancherel equality (Lem. 8.4) to complete the proof of the lemma. Returning to the proof of Proposition 8.1, Lemma 8.5 now allows us to write ||P m (A ⊗ B ⊗ C ⊗ D)||22 nχ Z XZ X

m 
2 b bχ ei (x)], e j 2 dλTd (w) ds dµ(x) = |b B(s)|2 | D(w)| Pγˆ [A(·)C X

b χ ∈G

Td ×R

b 22 ||A||2α ≤ || b B||22 || D||

i, j=1



sup

\ γˆ ∈supp( B⊗C⊗D)

 bχ |2 |||P m |||2α , n 2χ |C γˆ

(8.3)

where ||A||α denotes the usual norm on the space C α (X ) of α-H¨older continuous functions and |||P|||α denotes the operator norm on that space. LEMMA 8.6 Assume that the relation ∼ is weakly aperiodic. Then, for any γˆ bounded away from I in the dual space, we have that the spectral radius of Pγˆ : C α (X, C N ) → C α (X, C N ) is strictly less than 1.

Proof The proof is essentially a simple reinterpretation of the results from [28]. The idea is that the operator Pγˆ is quasi-compact with spectral radius at most 1. In particular, this means that if Pγˆ does not have spectral radius strictly smaller than 1, then there is an eigenfunction, Pγˆ h = eiξ h, say. However, h then satisfies (7.1), violating the weak aperiodicity hypothesis. This completes the proof of Lemma 8.6. In particular, Lemma 8.6 tells us that if we fix a neighborhood U ⊂ R ×\ G × Zd of I, then there exist C > 0 and 0 < ρ < 1 such that for any γˆ 6∈ U we have m ||Pγm ˆ || ≤ Cρ .

(8.4)

378

LEDRAPPIER and POLLICOTT

From (8.1), (8.3), and (8.4), we can deduce the following. 8.7 Assume that A is H¨older continuous and that F∗ ∈ L 1 (R× G ×Zd )∩ L 2 (R× G ×Zd ) has a Fourier transform c F∗ with compact support which does not contain an open ¯ is aperiodic. neighborhood U containing I. Further, assume that the relation ∼ on 3 m Then ||Pγˆ A ⊗ F∗ ||1 → 0 as m → +∞. LEMMA

Finally, we observe that, for any F ∈ L 1 (X × R × G × Zd ) satisfying Z F(x, t, g, z) dλR×G×Zd (t, g, z) = 0 for a.e. x, we can L 1 -approximate N

X

Ak ⊗ Fk < ,

F − k=0

1

say, where Fˆk has compact support, not containing I, and where Ak is an α-H¨older continuous function. Since we can apply (8.3) to the approximating function and since P does not expand the L 1 -norm, we see that Lemma 8.7 applies. By Lemma 8.2, this completes the proof of Proposition 8.1. b) → St1 (V b) In [30], M. Rees showed that for d = 1 or 2 the geodesic flow b gt : St1 (V is ergodic. This result has a natural analogue for frame flows, given by the following corollary to Proposition 8.1. COROLLARY 8.8 b) → Stn+1 (V b) is ergodic. If d = 1 or 2, then the frame flow b f t : Stn+1 (V

Proof We know by Proposition 8.1 that the symbolic model σ¯ tr for the frame flow is exact. Moreover, since b f t is a compact extension (by the group SO(n)) of the geodesic flow b gt and since we know from [30] and [17] that the geodesic flow b gt is recurrent, we can deduce that b f t is also recurrent. A straightforward application of Guivarc’h’s argument from [17, §2.4, (2)] shows that ergodicity of b f t is a consequence of exactness and recurrence. 9. The orbits of 0¯ It is particularly interesting to consider the case of normal subgroups 0 ⊂ 0 such that 0\0 = Zd . The analogue of Theorem 1 is somewhat more delicate. We restrict to the case of SL(2, C) for simplicity.

ERGODIC PROPERTIES OF LINEAR ACTIONS

379

THEOREM 5 Given 0 ⊂ 0 as above, there exists a function A : C2∗ × R+ → R+ satisfying

lim sup T →∞

A(z, T ) T 2 /(log T )d/2

=

C |z|2

such that for any compactly supported function f : C2∗ → R we have P Z ¯ |
where g : R → R is compactly supported (and thought of as approximating an indiP cator function), and q(z) = α cα z α is a polynomial on the unit circle. Symbolically, i is a state in the subshift of finite type (or, more generally, we introduce a cylinder for better approximations) representing a piece of unstable manifold for the geodesic flow, and the point x represents a piece of stable manifold for the geodesic flow.

380

LEDRAPPIER and POLLICOTT

Following [23], one can write N (T, e−P(v) g) = e−P(v)

X

cα

α

∞ X X

ehv, f

n (y)i−P(v)r n (y)+iα2n (y)

n=0 σ n y=x


× χi (y)g r n (y) + s − T δ f n (y)+a−b ∞ X Z X X n n n = cα ehv, f (y)i−P(v)r (y)+iα2 (y) α

n=0 σ n y=x

R×Td

 it (r n (y)−T ) +iα2n (y) × χi (y)g(t)e ˆ e dt dw ∞  X X Z i(t T −hw,bi) (L−(P(v)+it)r +hv+iw, f i+iα2 )n χi dt dw. = cα g(t)e ˆ α

R×Td

n=0

For the term α = 0, the asymptotics are worked out in [23]. For α 6= 0, we need the following additional observation (which is a simplified version of Lem. 8.6). 9.1 The operator L−(P(v)+it)r +hv+iw, f i+iα2 has unity as an eigenvalue if and only if α = 0, t = 0, and w = 0. In particular, the contribution to the above asymptotic is negligible. LEMMA

Proof The proof of the lemma follows by observing that the weighting is never cohomologous to a constant, except in the degenerate case. This should be compared with the proof of Proposition 7.1. One considers identities on closed orbits, and one eliminates the length by considering the flip map. Thus Theorem 5 is proved. We conclude this section with a second result that is also a straightforward generalization of results for SL(2, R) (see [6], [21]). 9.2 Assume that 0 ⊂ 0 satisfies 0/ 0 = Zd . We have the following: (1) there exist uncountably many different ergodic 0-invariant measures; (2) there exists v ∈ C2∗ such that the closure of the 0v does not contain 0 ∈ C2 . PROPOSITION

ERGODIC PROPERTIES OF LINEAR ACTIONS

381

Proof For part (1), we have the geometric interpretation that the 0-invariant measures corb). These, in respond to the transverse measures for the stable manifolds on Stn+1 (V ¯ turn, correspond to measures on 3 of the form  dt  × dλ S O(n) × dz, ehα,zi dν × R r dν where ν is the equilibrium state for −hr + hα, f i for α ∈ Zd . For part (2), we can choose v to correspond to a point in a direction corresponding to a nonhorospherical point in the boundary of ∂H3 . The result then follows by the arguments in [21, §1]. 10. The case of variable curvature We consider the generalization of Proposition 1.2 to some cases where the manifold V has variable sectional curvatures. (We can now allow V to be nonorientable, in which case f t : Stn+1 (V ) → Stn+1 (V ) denotes the flow on all orthonormal frames, irrespective of orientation, and is an O(n)-extension of the geodesic flow.) To begin, we need an extension of Proposition 1.1. 10.1 The frame flow f t : Stn+1 (V ) → Stn+1 (V ) associated to a compact manifold V is exact (i) if n is even but not equal to 6, or (ii) if n is odd but not equal to 7 and the sectional curvatures are pinched between −1 and −0.93 . . . , or (iii) if n = 6 or n = 7 and the sectional curvatures are pinched between −1 and −0.999785 . . . . PROPOSITION

Proof Part (i) was shown by M. Brin and M. Gromov [11]. Part (ii) was shown by Brin and H. Karcher [12]. Finally, part (iii) was proved in [13]. We have the following result for the strong stable manifolds for f t : Stn+1 (V ) → b) → Stn+1 (V b). Stn+1 (V ) and its Zd -cover b f t : Stn+1 (V 6 b) Assume that V satisfies the hypothesis of Proposition 10.1. The foliations of Stn+1 (V and Stn+1 (V ) by their strong stable manifolds are ergodic. THEOREM

382

LEDRAPPIER and POLLICOTT

Proof Let us assume, for simplicity, that V is orientable; the results in the nonorientable case then follow easily. Following the lines of the proof of Proposition 1.2, we can model f : Stn+1 (V ) → Stn+1 (V ) by the symbolic flow b σ r : X rA × SO(n) × Zd → X rA × d ¯ with an equivalence relation SO(n) × Z and we can define a corresponding space 3 ∼, as in §7. The conclusion of the theorem again follows from showing that this equivalence relation is weakly aperiodic. However, in the case of variable curvature, a more geometric interpretation of cross-ratios is required. ˜ denote the covering group for M, acting on the universal Let 0 ⊂ Isom( M) ˜ cover M for M. Let x, y, u, v be four distinct points on the boundary. We let γ˜y,x and γ˜v,u be the unique geodesics on M˜ such that γ˜y,x (−∞) = y, γ˜y,x (+∞) = x and γ˜v,u (−∞) = v, γ˜v,u (+∞) = u. In addition, we let γ˜y,u and γ˜v,x denote the unique geodesics with γ˜y,u (−∞) = y, γ˜y,u (+∞) = u and γ˜v,x (−∞) = v, γ˜v,x (+∞) = x. ˜ and fix a frame z. Consider the unit tangent vector γ˙ A (0) ∈ S M, (1) First, we can horospherically transport this frame z over the stable horosphere based at x to the unique frame z 1 such that the unit tangent vector is tangent to the geodesic γu,x and d( f˜t z, f˜t z 1 ) → 0, as t → +∞. (2) Second, we can horospherically transport the frame z 1 over the unstable horosphere based at u to the unique frame z 2 such that the unit tangent vector is tangent to the geodesic γu,v and d( f˜t z 1 , f˜t z 2 ) → 0, as t → −∞. (3) Next, we can horospherically transport z 2 over the stable horosphere based at v to the unique frame z 3 such that the unit tangent vector is tangent to the geodesic γ y,v and d( f˜t z 2 , f˜t z 3 ) → 0, as t → +∞. (4) Finally, we can horospherically transport z 3 over the unstable horosphere based at y to the unique frame z 4 such that the unit tangent vector is again tangent to the geodesic γ y,x and d( f˜t z 3 , f˜t z) → 0, as t → +∞. We can write z 4 := g( f˜T z 0 ), where g ∈ SO(n). Then e T can be considered as the absolute cross-ratio |x, y, u, v| (cf. [27]) and g plays the role of the angular crossratio of the points (x, y, u, v). We now proceed as follows. We say that (x, s) and (x 0 , s 0 ) ∈ X rA are stably equivalent (denoted (x, s) ∼s (x 0 , s 0 )) if there exist n, n 0 ≥ 0 such that xn+k = xn0 0 +k

for all k ≥ 0,

n0

r n (x) − s = r (x 0 ) − s 0 , and we say that (y, u) and (y 0 , u 0 ) are unstably equivalent (denoted (x, s) ∼u (x 0 , s 0 )) if there exist m, m 0 ≥ 0 such that 0 y−m−k = y−m 0 −k m0

r m (σ −m y) − u = r (σ

−m 0

for all k ≥ 0, 0

y 0 ) − u 0 + r u (σ −m−1 y, σ −m −1 y 0 ),

ERGODIC PROPERTIES OF LINEAR ACTIONS

383

v

z2 z3 u z1

z4

x

z0 y

Figure 2. The geometric interpretation of cross-ratios

P − p y) − r (σ − p z)). Assume that there are four points where r u (y, z) = p≥0 (r (σ (x (i) , si ) ∈ X rA , i = 0, 1, 2, 3, and that there is a number T such that (x (0) , s0 ) ∼s (x (1) , s1 ),

(x (1) , s1 ) ∼u (x (2) , s2 ),

(x (2) , s2 ) ∼s (x (3) , s3 ),

(x (3) , s3 ) ∼u σTr (x (0) , s0 ).

For each such quadruple, we have numbers m 0 , n 0 , n 1 , m 1 , m 2 , n 2 , n 3 , m 3 in the definition of ∼s , ∼u . We can then identify     T = r n 0 (x (0) ) − r n 1 (x (1) ) − r m 1 (σ −m 1 x (1) ) − r m 2 (σ −m 2 x (2) )     + r n 2 (x (2) ) − r n 3 (x (3) ) − r m 3 (σ −m 3 x (3) ) − r m 0 (σ −m 0 x (0) ) + r u (σ −m 1 x (1) , σ −m 2 x (2) ) + r u (σ −m 3 x (3) , σ −m 0 x (0) ).

384

LEDRAPPIER and POLLICOTT

We can analogously define
−1 u −m (0) −m (3)
8 = R 2−m 0 (σ m 0 x (0) ) R (σ 0 x , σ 3 x )R 2m 3 (σ −m 3 x (3) )
−1 s n (3) n (2)
× R 2n 3 (x (3) ) R (σ 3 x , σ 2 x )R 2n 2 (x (2) )
−1 u −m (2) −m (1)
× R 2−m 2 (σ m 2 x (2) ) R (σ 2 x , σ 1 x )R 2m 1 (σ −m 1 x (1) )

−1 s n (1) n (0) × R 2n 1 (x (1) ) R (σ 1 x , σ 0 x )R 2n 0 (x (0) ) , where R(2n (σ −n x)) := R(2(x))R(2(σ −1 x)) · · · R(2(σ −(n−1) x)) ∈ SO(n), and we write

−1 R u (σ −m i x (i), σ −m j x ( j) ) := lim R 2m (σ −(m+m i ) x (i) ) R 2m (σ −(m+m j ) x ( j) ) , m→+∞

−1 n j ( j) s n i (i) R 2n (σ n j x ( j) ) , R (σ x , σ x ) := lim R 2n (σ ni x (i) ) n→+∞

M = m 0 − m 3 + n2 − n3 + m 1 − m 2 + n0 − n1 (M counts the number of passages in 6 A ), and     F = f n 0 (x (0) ) − f n 1 (x (1) ) − f m 1 (σ −m 1 x (1) ) − f m 2 (σ −m 2 x (2) )     + f n 2 (x (2) ) − f n 3 (x (3) ) − f m 3 (σ −m 3 x (3) ) − f m 0 (σ −m 0 x (0) ) . Applying (7.1) repeatedly, we can write h 0 (x (1) ) = lim e−ihθ, f

n 0 (x (0) )− f n 1 (x (1) )i

n→+∞

e−it (r

n 0 +n (x (0) )−r n 1 +n (x (1) ))

eiξ(n 0 −n 1 )


−1
× R 2n 1 +n (x (1) ) R 2n 0 +n (x (0) ) h 0 (x (0) ) = e−ihθ, f

n 0 (x (0) )− f n 1 (x (1) )i

× e−it (r

n 0 (x (0) )−r n 1 (x (1) )+r s (σ n 0 x (0) ,σ n 1 x (1) ))

eiξ(n 0 −n 1 )


−1 s n (1) n (0)
× R 2n 1 (x (1) ) R (θ 1 x , θ 0 x )R 2n 0 (x (0) ) h 0 (x (0) ). We can similarly relate h(x (1) ), h(x (2) ), and h(x (3) ), and then, combining these identities, we see that 8h(x (0) ) = eihθ,Fi eit T ei Mξ h(x (0) ). (10.1) Consider the frames z 0 , z 1 , z 2 , z 3 , and z 4 = g( f˜T z 0 ), as in Figure 2, and assume ˜ that they project to unit tangent vectors v (0) , v (1) , v (2) , v (3) , v (4) = g˜ T v (0) ∈ S M. Moreover, assume that we can arrange the sections (in the flow direction) so that the vectors v (0) , v (1) , v (2) , and v (3) are the images by the coding of the points (x (i) , si ) ∈ X rA , i = 0, 1, 2, 3, respectively, and that (x (3) , s3 ) ∼u σTr (x (0) , s0 ). Observe that this is automatically the case as soon as the unit tangent vectors v (0) , v (1) , v (2) , v (3) , and v (4) have a unique symbolic representation. Then we can identify 8 = R(g),

ERGODIC PROPERTIES OF LINEAR ACTIONS

385

F as the Frobenius element of the path of horospherical transports, and M as the (signed) number of times the sequence of geodesics from v (0) to v (4) crosses the Markov sections through x, u, v, and y successively. Equation (10.1) then says that there is a vector h := h(x (0) ) in the space of the representation R such that R(g)h = eihθ,Fi eit T ei Mξ h.

(10.2)

Fix z 0 such that the geodesic tangent to v (0) has a unique symbolic representation, and set h = h(x (0) ). Then (10.2) holds for any (g, T, M, F) associated to a picture such that the geodesics (u, x), (u, v), and (y, v) have a unique symbolic representation. Consider first the case when n ≥ 3. By continuity, we have that the vector h is an eigenvector for R(g) for all choices of (u, v). In particular, when we can arrange that T = 0, we can identify g as an element of the Brin group H ⊂ SO(n), generated by a horospherical translation around us-paths of arbitrary length. Thus far, we have considered only us-paths of length 4, but, if necessary, we can iterate the argument above so that (10.2) holds more generally. Under hypotheses (i) – (iii) of Proposition 10.1, it is the case that H = SO(n) (see [13, Prop. 4]). We therefore find that the vector h is an eigenvector for all R(g), g ∈ SO(n). As we observed in Proposition 7.1, for n ≥ 3, SO(n) does not have a one-dimensional representation, and so we can deduce that the representation is trivial. For n = 2, it follows from [10] and [13] that it suffices to consider us-paths of length 4 in a neighborhood of z. Choose v (0) on a periodic orbit with a unique symbolic representation and Frobenius element zero. Periodic orbits with a unique symbolic representation are dense in a neighborhood of v (0) . Choose v (1) , v (2) , and v (3) to be sufficiently close that they pass through the same section as v (0) . For these choices, we readily see that when we compute F and M at the symbolic level, they are given by F = 0 and M = 0. By continuity, we see that for us-paths in a neighborhood of v (0) , we have (10.2) with T = 0, F = 0, and M = 0, and we can deduce that the representation is trivial. In all cases, we deduce that (7.1) has only trivial solutions; that is, the equivalence relation is aperiodic. The rest of the proof follows exactly as in the constant curvature case. Remarks. (1) When V is a compact manifold with negative sectional curvature satisfying the hypotheses of Proposition 10.1, we can deduce that the foliation W fss of Stn+1 (V ) by the strong stable manifolds is actually uniquely ergodic. More precisely, for any T transversal to the strong stable foliations Wgss for the geodesic flow on Stn+1 (V ), we can define a T 0 = T × SO(n) transversal to W fss . Any transverse measure µT 0 for W fss on T 0 must project to the unique transverse measure µT for Wgss on T , uniqueness being known by a result of Bowen and B. Marcus [8]. It remains to

386

LEDRAPPIER and POLLICOTT

show that dµT 0 = dµT × dλSO(n) , which can be deduced by the same general method as in the proof of Proposition 8.1. (2) For a compact K¨ahler manifold V of complex dimension m, we can consider the natural frame flow f t : Scm (V ) → Scm (V ) on the space of unitary frames Scm (V ). By [11, Th. 1.3], a sufficient condition for this flow to be ergodic is that either m is an odd number or m = 2. Moreover, it is shown that the Brin group is equal to U (m − 1). The proof of Theorem 6 generalizes to this setting, except that in the final step we do not have the convenience of U (m − 1) being free of one-dimensional representations. So we use both previous arguments. We first deduce from (10.2) that the representation R is one-dimensional, and then it suffices to consider us-paths in a neighborhood of some v (0) . Acknowledgments. We thank Franc¸oise Dal’bo for useful conversations. References [1]

[2]

[3] [4] [5]

[6]

[7] [8] [9] [10]

J. AARONSON and M. DENKER, “On exact group extensions” in Ergodic Theory and

Harmonic Analysis (Mumbai, 1999), Sankhy¯a Ser. A 62 (2000), 339 – 349. MR 2001m:37011 374 L. V. AHLFORS, “M¨obius transformations and Clifford numbers” in Differential Geometry and Complex Analysis, Springer, Berlin, 1985, 65 – 73. MR 86g:20065 362, 363, 365, 371 , “Cross-ratios and Schwarzian derivatives in Rn ” in Complex Analysis, Birkh¨auser, Basel, 1988, 1 – 15. MR 90a:30055 371, 373 L. V. AHLFORS and P. LOUNESTO, Some remarks on Clifford algebras, Complex Variables Theory Appl. 12 (1989), 201 – 209. MR 90m:15044 363 D. V. ANOSOV, Geodesic Flows on Closed Riemann Manifolds with Negative Curvature, Proc. Steklov Inst. Math. 90, Amer. Math. Soc., Providence, 1969. MR 39:3527 M. BABILLOT and F. LEDRAPPIER, “Geodesic paths and horocycle flow on abelian covers” in Lie Groups and Ergodic Theory (Mumbai, 1996), Tata Inst. Fund. Res. Stud. Math. 14, Tata Inst. Fund. Res., Bombay, 1998, 1 – 32. MR 2000e:37029 380 R. BOWEN, Symbolic dynamics for hyperbolic flows, Amer. J. Math. 95 (1973), 429 – 460. MR 49:4041 368 R. BOWEN and B. MARCUS, Unique ergodicity for horocycle foliations, Israel J. Math. 26 (1977), 43 – 67. MR 56:9594 385 R. BOWEN and D. RUELLE, The ergodic theory of Axiom A flows, Invent. Math. 29 (1975), 181 – 202. MR 52:1786 368 M. BRIN, “Ergodic theory of frame flows” in Ergodic Theory and Dynamical Systems (College Park, Md., 1979/1980), II, Progr. Math. 21, Birkh¨auser, Boston, 1982, 163 – 183. MR 83m:58059 385

ERGODIC PROPERTIES OF LINEAR ACTIONS

387

[11]

M. BRIN and M. GROMOV, On the ergodicity of frame flows, Invent. Math. 60 (1980),

[12]

M. BRIN and H. KARCHER, Frame flows on manifolds with pinched negative curvature,

[13]

K. BURNS and M. POLLICOTT, Stable ergodicity and frame flows, to appear in Geom.

[14]

R. ELLIS and W. PERRIZO, Unique ergodicity of flows on homogeneous spaces, Israel J.

[15]

J. M. FRANKS, Knots, links and symbolic dynamics, Ann. of Math. (2) 113 (1981),

[16]

H. FURSTENBERG, “The unique ergodicity of the horocycle flow” in Recent Advances

1 – 7. MR 81k:58053 381, 386 Compositio Math. 52 (1984), 275 – 297. MR 85m:58142 355, 381 Dedicata. 381, 385 Math. 29 (1978), 276 – 284. MR 57:12774 366 529 – 552. MR 83h:58074 372

[17]

[18] [19] [20] [21]

[22] [23]

[24] [25]

[26] [27]

[28]

in Topological Dynamics (New Haven, Conn., 1972), Lecture Notes in Math. 318, Springer, Berlin, 1973, 95 – 115. MR 52:14149 354 Y. GUIVARC’H, Propri´et´es ergodiques, en mesure infinie, de certains syst`emes dynamiques fibr´es, Ergodic Theory Dynam. Systems 9 (1989), 433 – 453. MR 91b:58190 370, 374, 378 G. A. HEDLUND, Fuchsian groups and mixtures, Ann. of Math. (2) 40 (1939), 370 – 383. 353 T. JØRGENSEN, Compact 3-manifolds of constant negative curvature fibering over the circle, Ann. of Math. (2) 106 (1977), 61 – 72. MR 56:8840 358 I. KIM, Ergodic theory and rigidity on the symmetric space of non-compact type, Ergodic Theory Dynam. Systems 21 (2001), 93 – 114. MR 2002c:37034 373 F. LEDRAPPIER, Horospheres on abelian covers, Bol. Soc. Brasil. Mat. (N.S.) 28 (1997), 363 – 375, MR 99g:58098a; Erratum, Bol. Soc. Brasil. Mat. (N.S.) 29 (1998), 195. MR 99g:58098b 380, 381 , Distribution des orbites des r´eseaux sur le plan r´eel, C. R. Acad. Sci. Paris S´er. I Math. 329 (1999), 61 – 64. MR 2000c:22009 354, 364, 365, 366, 379 , “Ergodic properties of some linear actions” in Geometry and Topology (Moscow, 1998), Vol. 7 (in Russian), Itogi Nauki Tekh. Ser. Sovrem. Mat. Prilozh. Temat. Obz. 68, Vseross. Inst. Nauchn. i Tekhn. Inform. (VINITI), Moscow, 1999, 87 – 113. MR 2003a:00043 353, 354, 380 “R. LIPSCHITZ,” Correspondence, Ann. of Math. (2) 69 (1959), 247 – 251. MR 20:7066 362 G. W. MACKEY, Unitary Group Representations in Physics, Probability, and Number Theory, 2d ed., Adv. Book Class., Addison-Wesley, Redwood City, Calif., 1989. MR 90m:22002 375 A. NOGUEIRA, Orbit distribution on R2 under the natural action of S L(2, Z), preprint, 2000. 354, 365 J.-P. OTAL, Sur la g´eometrie symplectique de l’espace des g´eod´esiques d’une vari´et´e a` courbure n´egative, Rev. Mat. Iberoamericana 8 (1992), 441 – 456. MR 94a:58077 382 W. PARRY and M. POLLICOTT, Zeta functions and the periodic orbit structure of hyperbolic dynamics, Ast´erisque 187 – 188, Soc. Math. France, Montrouge, 1990. MR 92f:58141 369, 377

388

LEDRAPPIER and POLLICOTT

[29]

M. RATNER, Markov partitions for Anosov flows on n-dimensional manifolds, Israel

[30]

M. REES, Checking ergodicity of some geodesic flows with infinite Gibbs measure,

[31]

R. SHARP, Closed orbits in homology classes for Anosov flows, Ergodic Theory

[32]

A. TERRAS, Harmonic Analysis on Symmetric Spaces and Applications, I, Springer,

J. Math. 15 (1973), 92 – 114. MR 49:4042 368 Ergodic Theory Dynam. Systems 1 (1981), 107 – 133. MR 83g:58037 378 Dynam. Systems 13 (1993), 387 – 408. MR 94g:58169 370 Berlin, 1985. MR 87f:22010 365

Ledrappier ´ Centre de Math´ematiques, Ecole Polytechnique, Unit´e Mixte de Recherche 7640 du Centre National de la Recherche Scientifique, F-91128 Palaiseau CEDEX, France; [email protected] Pollicott Department of Mathematics, University of Manchester, Oxford Road, Manchester M13 9PL, United Kingdom; [email protected]

MEROMORPHIC CONTINUATION OF THE SPECTRAL SHIFT FUNCTION VINCENT BRUNEAU and VESSELIN PETKOV

Abstract We obtain a representation of the derivative of the spectral shift function ξ(λ, h) in the framework of semiclassical “black box” perturbations. Our representation implies a meromorphic continuation of ξ(λ, h) involving the semiclassical resonances. Moreover, we obtain a Weyl-type asymptotics of the spectral shift function, as well as a Breit-Wigner approximation in an interval (λ − δ, λ + δ), 0 < δ < h. 1. Introduction The purpose of this paper is to obtain a meromorphic continuation of the derivative of the spectral shift function ξ(λ, h). This problem is closely related to the trace formulae (see [14], [34], [35] [22], [23], [30], [28], [29]) and to resonance expansions (see [8], [32]). For compact perturbations, the function ξ(λ, h) coincides with the scattering phase 1 log det S(λ, h), λ ∈ R, σ (λ, h) = 2πi where S(λ, h) = I + A(λ, h) : L 2 (S n−1 ) −→ L 2 (S n−1 ) is the scattering operator. For more information about the spectral shift function, we refer to [33]. In the classical case of h = 1, the first result proving a representation of σ (λ) = σ (λ, 1) containing the resonances z j ∈ C− = {z ∈ C : Im z < 0} was established by R. Melrose [17] for obstacle scattering in odd dimensions n ≥ 3. More precisely, given a function χ (t) ∈ C ∞ (R) such that 0 ≤ χ(t) ≤ 1, χ(t) = 1 for t ≤ 2, χ(t) = 0 for t ≥ 3, Melrose showed that σ (λ) = σsing (λ) + σreg (λ) with d 1 X  |z j |  Im z j σsing (λ) = − χ , dλ π λ |λ − z j |2

σsing (0) = 0, λ ∈ R,

j

σreg (λ) ∈ S n (R). DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 3, Received 21 September 2001. Revision received 10 December 2001. 2000 Mathematics Subject Classification. Primary 35P25; Secondary 35B34.

389

390

BRUNEAU and PETKOV

Since σ (λ, h) is the logarithmic derivative of the scattering determinant
s(λ, h) = det I + A(λ, h) , it is natural to examine the behavior of s(z, h) for z in the “physical half-plane,” where we have no resonances. This idea was developed by L. Guillop´e and M. Zworski [14] for the analysis of the scattering resonances for certain Riemann surfaces, and in the classical case of h = 1, Zworski [34], [35] gave an elegant proof of the trace formula for “black box” compact perturbations based on the meromorphic continuation of s(z, 1) (see [34] for other works on trace formulae). In [22] and [23], the Breit-Wigner approximation for the scattering phase was justified for “black box” scattering with compact perturbations in the classical and the semiclassical cases. Among the ideas introduced in [22] and [23], one of the main points in [23] was the estimate of the holomorphic function g(z, h), #

|g(z, h)| ≤ C()h −n ,

n # ≥ n,

(1.1)

in the local factorization s(z, h) = e g(z,h)

P(z, h) , P(z, h)

z ∈ ,

where P(z, h) =

Y

(z − w),

w∈Res L(h)∩ Im w6=0

 = (a, b) + i(−c, c),

0 < a < b, c > 0,

 = {z ∈ C : d(, z) < },

 > 0.

Here L(h) is a compactly supported perturbation of the operator −h 2 1, 0 < h ≤ h 0 , and n # depends on the estimates of the number of the eigenvalues of the reference operator. The local factorization immediately implies ∂z σ (z, h) =

1 1 ∂z g(z, h) + 2πi 2πi

X w∈Res L(h)∩ Im w6=0



1 1  − , z−w z−w

z ∈ , (1.2)

and for λ ∈ (a, b) we obtain an analogue of the formula of Melrose mentioned above. Combining (1.2) with the Birman-Krein formula, one easily obtains the trace formula of [28] exploiting the meromorphic continuation of ∂z σ (z, h) in {z ∈ C : Im z ≤ 0} (see [23, Th. 1]). Moreover, a similar factorization has been established in [23] in domains λ + h with an improved estimate for the holomorphic function g(z, h).

MEROMORPHIC CONTINUATION

391

In the case of “black box” long-range perturbations, the existence of the scattering operator and that of the scattering determinant are far from apparent. In this direction J. Sj¨ostrand [28], [29] proposed powerful techniques based on the complex scaling operators, introduced in [30], and complex analysis. The scattering determinant is replaced by D(z, h) = det(I + K˜ (z)), where K˜ (z) is a trace class operator that is not uniquely determined and the resonances are the zeros of D(z, h). Applying the approach of Sj¨ostrand, J.-F. Bony [1], [2] established upper and lower bounds on the number of the semiclassical resonances in small domains, and the Breit-Wigner approximation was extended to long-range perturbations in [4]. For a pair of selfadjoint operators L j (h), j = 1, 2, satisfying some assumptions (see Sec. 2), the spectral shift function ξ(λ, h) is a distribution in D 0 (R) such that
hξ 0 (λ, h), f (λ)iD 0 (R),D (R) = trbb f (L 2 (h)) − f (L 1 (h)) , f (λ) ∈ C0∞ (R), where trbb is a generalized trace defined in Section 2. We denote by Res L j (h), j = 1, 2, the set of the resonances w ∈ C− of L j (h). In this work we are strongly inspired by the approach in [23], and our main goal is to obtain an analogue of (1.2) in the cases when a scattering determinant is not available. We show that the representation (1.2) remains true in the general case of semiclassical “black box” scattering, replacing σ 0 (λ, h) by the “regular part” h i2 X ξ 0 (λ, h) − δ(λ − w) , w∈Res L j (h)∩(a,b)

j=1

where here and throughout the paper we use the notation [a j ]2j=1 = a2 − a1 . Our principal result is the following. 1 Assume that L j (h), j = 1, 2, satisfy the assumptions of Section 2. Let  b ei]−2θ0 ,2θ0 [ ]0, +∞[, 0 < θ0 < π/2, be an open simply connected set, and let W b  be an open, simply connected, and relatively compact set that is symmetric with respect to R. Assume that J =  ∩ R+ , I = W ∩ R+ are intervals. Then for λ ∈ I we have the representation h X i2 X 1 − Im w δ(λ−w) , (1.3) ξ 0 (λ, h) = Im r (λ, h)+ + j=1 π π|λ − w|2 THEOREM

w∈Res L j ∩J

w∈Res L j ∩ Im w6=0

where r (z, h) = g+ (z, h) − g+ (z, h), g+ (z, h) is a function holomorphic in , and g+ (z, h) satisfies the estimate #

|g+ (z, h)| ≤ C(W )h −n , with C(W ) > 0 independent on h ∈ ]0, h 0 ].

z ∈ W,

(1.4)

392

BRUNEAU and PETKOV

Remarks. • The terms related to the resonances are measures. In fact, the resonances w, Im w < 0, are related to harmonic measures Z Im w 1 ωC− (w, E) = − dt, E ⊂ R = ∂C− , π E |t − w|2 while the resonances w ∈ R+ coincide with the embedded eigenvalues of L j (h), j = S 1, 2. Moreover, in a small neighborhood Uλ (h) of every λ ∈ I \ 2j=1 {λ ∈ R : λ ∈ σpp (L j (h))}, the derivative ξ 0 (λ, h) coincides with a real analytic function on Uλ (h). In particular, if we have no embedded positive eigenvalues of L j (h) in I , then ξ 0 (λ, h) is real analytic in I. • The representations of ξ 0 (λ, h) obtained in [25] and [6] involve the traces of the cut-off resolvents χ (L j − λ ∓ i0)−1 χ, χ ∈ C0∞ (Rn ) and some regular terms whose meromorphic continuation is far from apparent. The form of ξ 0 (λ, h) in [25] and [6] has been used for the investigation of the Weyl-type asymptotics of ξ(λ, h) (see also [18], [5] for semiclassical asymptotics in the trapping case). The proof of (1.3) relies heavily on the work of Sj¨ostrand [29], while the arguments in [23] were self-contained and based on the semiclassical estimates of the scattering determinant. Having in mind (1.3), we obtain several results in the general case of “black box” semiclassical scattering. (I) We establish a Weyl-type asymptotics of the spectral shift function in the general framework of semiclassical “black box” perturbations, improving our previous result [6] and working without any assumption on the behavior of the resonances close to the real axis. We generalize the results of T. Christiansen [9] for compact perturbations and those of D. Robert [25] for long-range perturbations. Theorem 1 allows us to consider the sum of the harmonic measures related to the resonances w, Im w 6= 0, as a monotonic function and to apply a Tauberian argument as in [17]. (II) We present a new, direct, and short proof of the recent result of Bony and Sj¨ostrand [4] on the Breit-Wigner approximation in the long-range case (see Th. 3). For this purpose, the Weyl-type asymptotics obtained in Theorem 2 plays an essential role. Moreover, Theorems 2 and 3 are established under the “black box” assumptions in Section 2 and the condition (5.1). Thus we have a unified approach to these problems. Next, assuming the existence of a free resonance domain, we obtain a BreitWigner approximation involving only the resonances w lying in small “boxes”  w ∈ C : | Re w − λ| ≤ R(h), | Im w| ≤ R1 (h) √ with R(h) = h R1 (h) = O (h ∞ ). (III) In the same way as in [23], we obtain the local trace formula of Sj¨ostrand [28], [29] in a slightly stronger version (see Sec. 7). Moreover, we prove a trace for-

MEROMORPHIC CONTINUATION

393

mula involving the unitary groups e−i(t/ h)L j (h) , j = 1, 2 (see Th. 5), which is a semiclassical version of the classical trace formulae. We expect that the approach of our work could be useful in other situations, as in the analysis of periodic potentials (see [11]) or the study of matrix Schr¨odinger operators (see [19]) if a representation like (1.3) is established. The plan of the paper is the following. In Section 2 we introduce the “black box” scattering assumptions, and in Section 3 we obtain a formula for ξ 0 (λ, h) involving the limits of the functions σ± (z) as Im z → 0. Theorem 1 is proved in Section 4, and in Section 5 we establish a Weyl-type asymptotics for the spectral shift function ξ(λ, h). The semiclassical Breit-Wigner approximation is established in Section 6 together with a stronger approximation based on some recent results of P. Stefanov [31]. In Section 7 we prove some trace formulae combining (1.3) with the arguments of [23]. In particular, we obtain a trace formula involving the unitary groups e−i(t/ h)L j (h) . Finally, in Section 8 the Breit-Wigner approximation is applied to establish the existence of clusters of resonances close to the real axis. 2. Preliminaries We start with the abstract “black box” scattering assumptions introduced in [30], [28], and [29]. The operators L j (h) = L j , j = 1, 2, 0 < h ≤ h 0 , are defined in domains D j ⊂ H j of a complex Hilbert space H j with an orthogonal decomposition
H j = H R0 , j ⊕ L 2 Rn \ B(0, R0 ) , B(0, R0 ) = {x ∈ Rn : |x| ≤ R0 },

R0 > 0, n ≥ 2.

Below, h > 0 is a small parameter, and we suppose that the assumptions are satisfied for j = 1, 2. We suppose that D j satisfies
1lRn \B(0,R0 ) D j = H 2 Rn \ B(0, R0 ) (2.1) uniformly with respect to h in the sense of [28]. More precisely, equip H 2 (Rn \ B(0, R0 )) with the norm khh Di2 uk L 2 , hh Di2 = 1 + (h D)2 , and equip D j with the norm k(L j + i)ukH j . Then we require that 1lRn \B(0,R0 ) : D j −→ H 2 (Rn \ B(0, R0 )) be uniformly bounded with respect to h and that this map have a uniformly bounded right inverse. Assume that 1l B(0,R0 ) (L j + i)−1 is compact (2.2) and that
(L j u)|Rn \B(0,R0 ) = Q j u|Rn \B(0,R0 ) , where Q j is a formally self-adjoint differential operator X Q ju = a j,ν (x; h)(h Dx )ν u |ν|≤2

(2.3)

(2.4)

394

BRUNEAU and PETKOV

with a j,ν (x; h) = a j,ν (x) independent of h for |ν| = 2 and a j,ν ∈ Cb∞ (Rn ) uniformly bounded with respect to h. We also assume the following properties. There exists C > 0 such that X l j,0 (x, ξ ) = a j,ν (x)ξ ν ≥ C|ξ |2 , (2.5) |ν|=2

X

a j,ν (x; h)ξ ν −→ |ξ |2 ,

|x| −→ ∞,

(2.6)

|ν|≤2

uniformly with respect to h. There exists n > n such that we have |a1,ν (x; h) − a2,ν (x; h)| ≤ O (1)hxi−n

(2.7)

uniformly with respect to h. This assumption guarantees that for every f ∈ C0∞ (R) the operator f (L 1 ) − f (L 2 ) is “trace class near ∞.” There exist θ0 ∈ ]0, π/2[,  > 0, and R1 > R0 , so that the coefficients a j,ν (x; h) of Q j can be extended holomorphically in x to  0 = r ω; ω ∈ Cn , dist(ω, S n−1 ) < , r ∈ C, r ∈ ei[0,θ0 ] ]R1 , +∞[ , (2.8) and (2.6) and (2.7) extend to 0. ˜ n , and let R˜ > 2R. Set Let R > R0 , let TR˜ = (R/ RZ)
H j# = H R0 , j ⊕ L 2 TR˜ \ B(0, R0 ) , and consider a differential operator Q #j =

X

a #j,ν (x; h)(h D)ν

|ν|≤2

on TR˜ with a #j,ν (x; h) = a j,ν (x; h) for |x| ≤ R satisfying (2.3), (2.4), and (2.5) with Rn replaced by TR˜ . Consider a self-adjoint operator L #j : H j# −→ H j# defined by L #j u = L j ϕu + Q #j (1 − ϕ)u,

u ∈ D #j ,

with domain  D #j = u ∈ H j# : ϕu ∈ D j , (1 − ϕ)u ∈ H 2 , where ϕ ∈ C0∞ (B(0, R); [0, 1]) is equal to 1 near B(0, R0 ). Denote by N (L #j , [−λ, λ]) the number of eigenvalues of L #j in the interval [−λ, λ]. Then we assume that   #  λ n j /2 N (L #j , [−λ, λ]) = O , n #j ≥ n, λ ≥ 1. (2.9) h2

MEROMORPHIC CONTINUATION

395

Finally, we suppose that with some constant C ≥ 0 independent on h we have sp L j (h) ⊂ [−C, ∞[,

j = 1, 2,

(2.10)

where sp(L) denotes the spectrum of L . This condition is a technical one, and we expect that by a finer version of Proposition 1 we could cover the general case. Given f ∈ C0∞ (R) independent on h and χ ∈ C0∞ (Rn ) equal to 1 on B(0, R0 ), we can define trbb [ f (L j )]2j=1 , as in [28] and [29], by the equality
 2 trbb f (L 2 ) − f (L 1 ) = tr(χ f (L j )χ + χ f (L j )(1 − χ) + (1 − χ ) f (L j )χ) j=1  2 + tr (1 − χ) f (L j )(1 − χ) j=1 . Following [28] and [29], we can define the resonances w ∈ C− by the complex scaling method as the eigenvalues of the complex scaling operators L j,θ , j = 1, 2. We denote by Res L j (h), j = 1, 2, the set of resonances and set n # = max{n #1 , n #2 }. 3. Representation of the derivative of the spectral shift function Consider the resolvents Z ±∞ R j (λ ± i) = i eitλ e−it (L j ∓i) dt, λ ∈ R,  > 0, 0

R j (λ − i) = −i

Z

0

eitλ e−it (L j +i) dt.

−∞

Given a function f (λ) ∈ C0∞ (R), we have Z 1 R j (λ + i) f (λ) dλ = 2πi Z 1 − R j (λ − i) f (λ) dλ = 2πi

Z ∞ 1 fˆ(−t)e−it L j −t dt, 2π 0 Z 0 1 fˆ(−t)e−it L j +t dt, 2π −∞

where fˆ denotes the Fourier transform of f. Choose z 0 ∈ R− away from sp(L j ), j = 1, 2, and set g(λ) = (λ − z 0 )m f (λ), where the integer m > n/2 is taken sufficiently large and independent on h. Applying the above formula, we obtain Z  1 trbb (L j − z 0 )−m (λ + i − z 0 )m R j (λ + i) 2πi
2 − (λ − i − z 0 )m R j (λ − i) j=1 f (λ) dλ h Z ∞
1 = trbb (L j − z 0 )−m e−t−it L j g(−t) ˆ + iG +, (t) dt 2π 0 Z 0
i2 + et−t L j g(−t) ˆ + iG −, (t) dt . −∞

j=1

(3.1)

396

BRUNEAU and PETKOV

Here G ±, (t) are some functions in S (R) related to the Fourier transform of λk f (λ), 0 ≤ k ≤ m − 1, which are uniformly bounded with respect to 0 <  < 1. To justify the limit  ↓ 0 in (3.1), we need to establish the estimates of the trace uniformly with respect to  > 0. To do this, we prove the following. 1 For any t ∈ R, the trace trbb [(L j − z 0 )−m e−it L j ]2j=1 is well defined, and LEMMA

 2
# trbb (L j − z 0 )−m e−it L j j=1 = O h −n (1 + |t|) . Proof Let χ ∈ C0∞ (Rn ) be equal to 1 near B(0, R1 ), R1 > R0 . Since the operators χ (L j − z 0 )−m and (L j − z 0 )−m χ are trace class (see [28]) and since e−it L j is uniformly bounded with respect to t, it is clear that χ(L j −z 0 )−m e−it L j and (L j −z 0 )−m e−it L j χ # are trace class with trace bounded by O (h −n ). To be more precise, let us note that in [29] condition (2.10) is not assumed, and we can formally apply the results of [29] for z 0 ∈ C \ R. In our case of z 0 ∈ R− and according to the resolvent equation, we have
m (L j − z 0 )−m = (L j − z 1 )−m I + (z 0 − z 1 )(L j − z 0 )−1 . So taking z 1 ∈ C \ R, we obtain the trace class properties mentioned above. Now consider the operator  2 (1 − χ)(L j − z 0 )−m e−it L j (1 − χ) j=1 . By the Duhamel formula, we obtain (1 − χ )(L j − z 0 )−m e−it L j (1 − χ) = e−it Q j (1 − χ)(L j − z 0 )−m (1 − χ) Z t +i e−i(t−s)Q j [χ, L j ](L j − z 0 )−m e−is L j ds. 0 #

The integrand is a trace class operator with trace bounded by O (h −n ), and it remains to study the operator  −it Q  j (1 − χ)(L − z )−m (1 − χ) 2 e . j 0 j=1 For R1 > R0 , χ0 ∈ C0∞ (Rn ) equal to 1 near B(0, R1 ), and χ0 ≺ χ , we have (L j − z 0 )−1 (1 − χ) = (1 − χ0 )(Q j − z 0 )−1 (1 − χ) + (L j − z 0 )−1 [Q j , χ0 ](Q j − z 0 )−1 (1 − χ ).

MEROMORPHIC CONTINUATION

397

Here and below, the notation ϕ ≺ ψ means that ψ = 1 on supp ϕ. Choose cut-off functions θ N ≺ · · · ≺ θ1 ≺ χ so that θ N = 1 on B(0, R0 ), and apply the telescopic formula (L j −z 0 )−1 [Q j , χ0 ](Q j − z 0 )−1 (1 − χ) = (L j − z 0 )−1 [Q j , χ0 ](Q j − z 0 )−1 [Q j , θ N ](Q j − z 0 )−1 [Q j , θ N −1 ] × · · · × [Q j , θ1 ](Q j − z 0 )−1 (1 − χ). For N > n/2, this operator is trace class. In fact, for χ˜ ∈ C0∞ equal to 1 on supp θ N , the operator χ˜ (Q j − i)−N /2 (Q j − i) N /2 [Q j , θ N ](Q j − z 0 )−1 · · · [Q j , θ1 ](Q j − z 0 )−1 (1 − χ) is trace class, while (L j − z 0 )−1 [Q j , χ0 ](Q j − z 0 )−1 is bounded. Here we have used the fact that Q j are elliptic operators and that (Q j − z 0 )−1 = O (1) : H N (Rn ) −→ H N +2 (Rn ),

∀N ∈ N.

Repeating this procedure, modulo trace class operators we obtain e−it Q j (L j − z 0 )−m (1 − χ) = e−it Q j (1 − θm )(Q j − z 0 )−1 · · · (1 − θ1 )(Q j − z 0 )−1 (1 − χ). In the same way, since θk ≺ θk−1 , each term θk (Q j −z 0 )−1 (1−θk−1 ) in the above product is a trace class operator, and modulo a trace class operator we are going to study  −it Q  j (Q − z )−m (1 − χ) 2 e . j 0 j=1 Consider the difference (Q 2 − z 0 )−m e−it Q 2 − (Q 1 − z 0 )−m e−it Q 1
= e−it Q 2 (Q 2 − z 0 )−m − (Q 1 − z 0 )−m + (e−it Q 2 − e−it Q 1 )(Q 1 − z 0 )−m . For the first term on the right-hand side, observe that the operator (Q 2 − z 0 )−m − (Q 1 − z 0 )−m for m > n/2 is a trace class one (see [10], [24], [28]). To handle the second term, notice that Z t (e−it Q 2 − e−it Q 1 )(Q 1 − z 0 )−m = i e−i(t−s)Q 2 (Q 1 − Q 2 )(Q 1 − z 0 )−m e−is Q 1 ds, 0

and use the fact that (Q 1 − Q 2 )(Q 1 − z 0 )−m is trace class for m > n/2 + 1.

398

BRUNEAU and PETKOV

According to Lemma 1, in equation (3.1) we can take the limit  ↓ 0 with respect to the norm in the space of trace class operators, and taking into account the definition of trbb (·), we get h Z ∞
1 −m lim trbb (L j − z 0 ) e−t−it L j g(−t) ˆ + iG +, (t) dt ↓0 2π 0 Z 0
i2 + et−it L j g(−t) ˆ + iG −, (t) dt j=1 −∞ Z ∞ h i 2 1 = trbb (L j − z 0 )−m e−it L j g(−t) ˆ dt j=1 2π −∞   2 = trbb (L j − z 0 )−m g(L j ) j=1
= trbb f (L 1 ) − f (L 2 ) = hξ 0 (λ, h), f (λ)iD 0 (R),D (R) . Thus we have proved the following. PROPOSITION

1

We have ξ 0 (λ, h) =

 1 lim trbb (λ + i − z 0 )m (L j − λ − i)−1 2πi ↓0
2 −(λ − i − z 0 )m (L j − λ + i)−1 (L j − z 0 )−m j=1 , (3.2)

where the limit is taken in the sense of distributions D 0 (R). Introduce the functions  2 σ± (z) = (z − z 0 )m trbb (L j − z)−1 (L j − z 0 )−m j=1 ,

± Im z > 0,

(3.3)

which are well defined (see [29] and Prop. 2). The relation  2  2 trbb (L j − (λ − i))−1 (L j − z 0 )−m j=1 = trbb (L j − (λ + i))−1 (L j − z 0 )−m j=1 immediately implies σ− (z) = σ+ (z),

Im z < 0.

(3.4)

Equality (3.4) plays a crucial role in the proof of (1.3), and our choice of real z 0 is related to the above relation. 4. Meromorphic continuation of the spectral shift function In this section we prove our principal result, given in Theorem 1. Taking 0 < θ ≤ θ0 < π/2, consider the complex scaling operators L j,θ related to L j , j = 1, 2,

MEROMORPHIC CONTINUATION

399

introduced by Sj¨ostrand and Zworski (see [30], [28], and [29, Sec. 2]). More precisely, given 0 > 0, R1 > R0 , consider a function i πh × [0, ∞[ 3 (θ, t) 7→ C f θ (t) : 0, 2 which is an injection for every θ and has the following properties: f θ (t) = t

for 0 ≤ t ≤ R1 ,

0 ≤ arg f θ (t) ≤ θ,

∂t f θ 6= 0,

arg f θ (t) ≤ arg ∂t f θ (t) ≤ arg f θ + 0 , f θ (t) = eiθ t

for t ≥ T0 ,

where T0 depends on 0 and R1 . Next, consider the map κθ : Rn 3 x = tω 7→ f θ (t)ω ∈ Cn ,

t = |x|,

and introduce 0θ = κθ (Rn ), which coincides with Rn along B(0, R1 ). We define
H j,θ = H R0 , j ⊕ L 2 0θ \ B(0, R0 ) and L j,θ : H j,θ −→ H j,θ with domain D j as the operator L j,θ u = L j (χ1 u) + Q j |0θ (1 − χ1 )u, χ1 ∈ C0∞ (B(0, R1 )) being a function equal to 1 near B(0, R0 ). Let  b ei]−2θ,2θ [ ]0, +∞[ be a simply connected open relatively compact set such that  ∩ R+ = J is an interval. The spectrum of L j,θ outside of e−2iθ [0, +∞[ consists of the negative eigenvalues of L j and the eigenvalues in e−i[0,2θ [ ]0, +∞[ (see [28]). Since the spectrum of L j is bounded from below, we may choose z 0 ∈ R− , z 0 ∈ / , so that z 0 is away from sp(L j ) and sp(L j,θ ), j = 1, 2. Given a positive number δ > 0, we can apply [29, Prop. 4.1], saying that for all z ∈  ∩ {z : Im z ≥ δ} we have  2  2 trbb (L j − z)−1 (L j − z 0 )−m j=1 = trbb (L j,θ − z)−1 (L j,θ − z 0 )−m j=1 , (4.1) where in the definition of the complex scaling operators L j,θ the parameter 0 is chosen small enough. Notice that the choice of z 0 ∈ ei[30 , min(π, 2π −2θ −30 )] ]0, +∞[ in [29] says that we may take z 0 ∈ R− , assuming θ < π/2 − (3/2)0 . Below we assume δ and θ fixed, and in the notation L j we drop the index j, writing L · when the properties are satisfied for both operators L j , j = 1, 2. Following [29, Sec. 4], there exists an operator Lˆ ·,θ : D· −→ H· so that # K ·,θ = Lˆ ·,θ − L ·,θ has rank O (h −n ),

400

BRUNEAU and PETKOV

and for all N , M ∈ N we have K ·,θ = O (1) : D (L ·N ) −→ D (L ·M ). Second, K ·,θ is compactly supported; that is, if χ ∈ C0∞ (Rn ) is equal to 1 on B(0, R) for R ≥ R0 large enough, we have K ·,θ = χ K ·,θ χ. Finally, for every N ∈ N we have ( Lˆ ·,θ − z)−1 = O (1) : D (L ·N ) −→ D (L ·N +1 ) uniformly for z ∈ . These properties imply for z ∈ ∩{Im z > 0} the representation (L ·,θ − z)−1 = ( Lˆ ·,θ − z)−1 + (L ·,θ − z)−1 K ·,θ ( Lˆ ·,θ − z)−1 . The contributions related to the resolvent ( Lˆ ·,θ − lowing.

z)−1

(4.2)

are examined in the fol-

2 There exists a function a+ (z, h) holomorphic in  such that for z ∈  ∩ {Im z > 0} we have  2 σ+ (z) = tr (L j,θ − z)−1 K j,θ ( Lˆ j,θ − z)−1 j=1 + a+ (z, h). (4.3) PROPOSITION

Moreover, #

|a+ (z, h)| ≤ C()h −n ,

z ∈ ,

(4.4)

with a constant C() independent on h ∈ ]0, h 0 ]. Remark. The singularities of σ+ (z) for Im z ↓ 0 are independent on z 0 ∈ R− and m ∈ N. Proof According to (4.2), for z ∈  ∩ {Im z ≥ δ} we have  2 σ+ (z) = (z − z 0 )m trbb ( Lˆ j,θ − z)−1 (L j,θ − z 0 )−m j=1 (4.5) 
 2 + (z − z 0 )m tr (L j,θ − z)−1 K j,θ ( Lˆ j,θ − z)−1 (L j,θ − z 0 )−m j=1 . (4.6) From the resolvent equation we obtain (z − z 0 ) (L j,θ − z 0 ) m

−m

(L j,θ − z)

−1

= (L j,θ − z)

−1

−

m X

(z − z 0 )k−1 (L j,θ − z 0 )−k .

k=1

To treat (4.6), we use the cyclicity of the trace and the above equality and conclude that this term is equal to tr[(L j,θ − z)−1 K j,θ ( Lˆ j,θ − z)−1 ]2j=1 modulo a function #

holomorphic in  and bounded by O (h −n ).

MEROMORPHIC CONTINUATION

401

Now we pass to the analysis of (4.5). Our purpose is to show that (4.5) is holo# morphic in  and bounded by O (h −n ). By construction, ( Lˆ j,θ − z)−1 is holomorphic on , and for any cut-off function χ ∈ C0∞ (Rn ), χ = 1 on B(0, R0 ) with supp χ ⊂ B(0, R1 ), the operators χ(L j,θ − z 0 )−m , (L j,θ − z 0 )−m χ are trace class ones. Hence the function tr(( Lˆ j,θ − z)−1 (L j,θ − z 0 )−m χ) is holomorphic in . On the other hand, (L j,θ − z 0 )−m ( Lˆ j,θ − z)−1 − ( Lˆ j,θ − z)−1 (L j,θ − z 0 )−m = (L j,θ − z 0 )−m (L j,θ − z)−1 K j,θ ( Lˆ j,θ − z)−1 − (L j,θ − z)−1 × K j,θ ( Lˆ j,θ − z)−1 (L j,θ − z 0 )−m .

(4.7)

Consequently, for Im z > 0 if χ1 ∈ C0∞ (Rn ) is a cut-off function and χ1 ≺ χ , applying the cyclicity of the trace once more, we get
tr χ1 ( Lˆ j,θ − z)−1 (L j,θ − z 0 )−m (1 − χ) = 0. Thus it remains to examine  2 τ+ (z) = tr (1 − χ1 )( Lˆ j,θ − z)−1 (1 − χ)(L j,θ − z 0 )−m (1 − χ) j=1 . Consider the operator Q ·,θ = Q · |0θ , and note that for ψ ∈ C ∞ supported away from B(0, R1 ) we have L ·,θ ψ = Q ·,θ ψ. Repeating the construction of Lˆ ·,θ in [29, Sec. 4], we can find an operator Qˆ ·,θ : H 2 (0θ ) −→ L 2 (0θ ) so that Qˆ ·,θ − Q ·,θ has rank O (h −n ), the operator Qˆ ·,θ − Q ·,θ is compactly supported, and for z ∈  we have ( Qˆ ·,θ − z)−1 = O (1) : D(Q ·N ) −→ D(Q ·N +1 ),

∀N ∈ N.

Moreover, for ψ ∈ C ∞ supported away from B(0, R1 ), we have Lˆ ·,θ ψ = Qˆ ·,θ ψ, and for χ ∈ C0∞ (0θ ) equal to 1 on a sufficiently large set, z ∈ , and χ1 ≺ χ0 ≺ χ , we obtain ( Lˆ ·,θ − z)−1 (1 − χ ) = (1 − χ0 )( Qˆ ·,θ − z)−1 (1 − χ) + ( Lˆ ·,θ − z)−1 [ Qˆ ·,θ , χ0 ]( Qˆ ·,θ − z)−1 (1 − χ ). As above, we assume that z 0 ∈ R− is chosen so that z 0 ∈ / sp(Q j ), z 0 ∈ / sp(Q j,θ ), j = 1, 2. For simplicity of notation, below we omit the index θ , and we get  2 τ+ (z) = tr (1 − χ0 )( Qˆ j − z)−1 (1 − χ)(L j − z 0 )−m (1 − χ) j=1  + tr (1 − χ1 )( Lˆ j − z)−1 [ Qˆ j , χ0 ]( Qˆ j − z)−1 2 × (1 − χ)(L j − z 0 )−m (1 − χ) j=1 .

402

BRUNEAU and PETKOV

Obviously, [ Qˆ j , χ0 ] = [Q j , χ0 ] + M j with a trace class operator M j . To show that the operator [Q j , χ0 ]( Qˆ j − z)−1 (1 − χ) is a trace class one, we apply the telescopic formula, choosing cut-off functions θ N ≺ θ N −1 ≺ · · · ≺ θ1 ≺ χ , and we write [Q j , χ0 ]( Qˆ j − z)−1 (1 − χ) = [Q j , χ0 ]( Qˆ j − z)−1 χ(Q j − i)−m   × (Q j − i)m [ Qˆ j , θ N ]( Qˆ j − z)−1 [ Qˆ j , θ N −1 ] · · · [ Qˆ j , θ1 ]( Qˆ j − z)−1 (1 − χ ) with N ≥ 2m > n. The operator in the brackets [· · · ] and [Q j , χ0 ]( Qˆ j − z)−1 are bounded, while χ(Q j − i)−m is trace class. Thus the term involving [ Qˆ j , χ0 ] is # holomorphic in  and bounded by O (h −n ). As in the proof of Proposition 1, we have

(1 − χ )(L j − z 0 )−m (1 − χ) − (1 − χ)(Q j − z 0 )−m (1 − χ ) = O (h −n # ). tr Moreover, (Q j − z 0 )−m χ is trace class, and, consequently, there exists a function # b(z, h) holomorphic in  and bounded by O (h −n ) so that  2 τ+ (z) = b(z, h) + tr (1 − χ)( Qˆ j − z)−1 (Q j − z 0 )−m (1 − χ ) j=1 . (4.8) We write ( Qˆ 2 − z)−1 (Q 2 − z 0 )−m − ( Qˆ 1 − z)−1 (Q 1 − z 0 )−m   = ( Qˆ 2 − z)−1 (Q 2 − z 0 )−m − (Q 1 − z 0 )−m   + ( Qˆ 2 − z)−1 − ( Qˆ 1 − z)−1 (Q 1 − z 0 )−m = I + I I. According to [28] and [29], the operator (Q 2 − z 0 )−m − (Q 1 − z 0 )−m is a trace # class one, and the contribution of I is holomorphic and bounded by O (h −n ). For I I , we obtain the representation I I = ( Qˆ 2 − z)−1 ( Qˆ 1 − Qˆ 2 )( Qˆ 1 − z)−1 (Q 1 − z 0 )−m . It is clear that Qˆ 1 − Qˆ 2 = Q 1 − Q 2 + K 1,2 with a finite rank operator K 1,2 , and modulo a trace class operator we have

I I = ( Qˆ 2 − z)−1 (Q 1 − Q 2 )(Q 2 − z 0 )−m (Q 2 − z 0 )m ( Qˆ 1 − z)−1 (Q 1 − z 0 )−m . The second factor is a trace class operator, while the first and the third ones are bounded operators. Consequently, I I has the same property as I . Combining the # above results, we conclude that τ+ (z) is holomorphic in  and bounded by O (h −n ). To establish (4.3), notice that the right-hand side of this equality is holomorphic for z ∈  ∩ {Im z > 0}. The left-hand side is also holomorphic in this domain since we may apply (4.1) with different δ > 0, 0 > 0, and 0 < θ < π/2 − (3/2)0 . By analytic continuation, we deduce (4.3), and the proof of Proposition 2 is complete.

MEROMORPHIC CONTINUATION

403

Proof of Theorem 1 To obtain a meromorphic continuation of σ+ (z) through the real axis, it suffices to do this for the trace involving K j,θ . Next, we follow closely the argument of Sj¨ostrand [29], and since θ is fixed, we omit it in the notation. Setting K˜ · (z) = K · (z − Lˆ · )−1 , from [29, (4.31)] we get the representation  

−1 ∂ K˜ · (z) − tr (L · − z)−1 K · ( Lˆ · − z)−1 = tr 1 + K˜ · (z) ∂z
˜ = ∂z log det 1 + K · (z) , and the resonances of L · are precisely the zeros of the function
# D(z, h) = det 1 + K˜ · (z) = O (1) exp(Ch −n ).

(4.9)

Notice that the multiplicities of the resonances and the zeros coincide. In the notation below we omit the subscript “·” since the argument does not depend on j = 1, 2. Let Res(L) be the resonances of L, and let Y D(z, h) = G(z, h) (z − w), w∈Res(L)∩

where G(z, h) and 1/G(z, h) are holomorphic in  and the resonances in the product are repeated following their multiplicity. Obviously, X

∂z log D(z, h) = ∂z log G(z, h) +

w∈Res(L)∩

and according to the estimate [29, (4.54)], we get ∂ ˜ −n # , log G(z, h) ≤ C()h ∂z

1 , z−w

˜ z ∈ ,

(4.10)

˜ b  is an arbitrary open simply connected domain and C() ˜ is independent where  on h ∈ ]0, h 0 ]. Going back to representation (3.2) and taking into account (3.4), we observe that for λ ∈ I ⊂ R+ , Im w 6= 0, we have   1 1 1 − Im w − lim − = , 2πi ↓0 λ + i − w λ − i − w π|λ − w|2 while for w ∈ R we get −

  1 1 1 lim − = δ(λ − w), 2πi ↓0 λ + i − w λ − i − w

where both limits are taken in the sense of distributions. Combining Propositions 1 and 2 and the above arguments, we complete the proof of Theorem 1.

404

BRUNEAU and PETKOV

Representation (1.3) shows that modulo a constant the spectral shift function ξ(λ, h) coincides with the distribution Z λ i2 1h X | Im w| ξ(λ, h) = dµ 2 j=1 π λ0 |µ − w| w∈Res L j (h) Im w6=0

 2 + #{µ ∈ [λ0 , λ] : µ ∈ σpp (L j (h))} j=1 Z 1 λ + Im r (µ, h) dµ, λ0 > 0, λ0 ∈ / I. π λ0 S In particular, for λ ∈ I \ 2j=1 {λ ∈ R : λ ∈ sppp (L j (h))} the distribution ξ(λ, h) is continuous, and the function  2 η(λ, h) = ξ(λ, h) − #{µ ∈ [λ0 , λ] : µ ∈ sppp (L j (h))} j=1 is real analytic in I . 5. Weyl asymptotics In this section we obtain a Weyl-type asymptotics for the spectral shift function. We generalize the results of Christiansen [9] and Robert [25] covering the “black box” long-range perturbations of the Laplacian, and we improve our previous result (see [6, Th. 2]) working without any condition on the behavior of the resonances close to the real axis. We say that λ ∈ R is a noncritical energy level for Q if for all (x, ξ ) ∈ 6λ = {(x, ξ ) ∈ R2n : l(x, ξ ) = λ} we have ∇x,ξ l(x, ξ ) 6= 0, l(x, ξ ) being the principal symbol of Q. Given a Hamiltonian l(x, ξ ), denote by
exp(t Hl )(x0 , ξ0 ) = x(t, x0 , ξ0 ), ξ(t, x0 , ξ0 ) the trajectory of the Hamilton flow exp(t Hl ) passing through (x0 , ξ0 ) ∈ 6λ . Recall that λ ∈ J is a nontrapping energy level for l(x, ξ ) if for every R > 0 there exists T (R) > 0 such that for (x0 , ξ0 ) ∈ 6λ , |x0 | < R, the x-component of the trajectory of exp(t Hl ) passing through (x0 , ξ0 ) satisfies |x(t, x0 , ξ0 )| > R,

∀|t| > T (R).

Denote by N (L #j , I ) the number of eigenvalues of L #j in the interval I . From assumptions (2.5) and (2.10), we easily deduce that there exists a constant C # such that the spectrums of L #j , j = 1, 2, do not intersect the interval ] − ∞, −C # ], and consequently N (L #j , ]− ∞, −C # ]) = 0. In fact, let χ0 , χ , χ1 ∈ C0∞ (B(0, R); [0, 1])

MEROMORPHIC CONTINUATION

405

be equal to 1 on B(0, R0 ), and let χ1  χ  χ0 . Using the resolvent equality, we get (L #j − z)−1 = (L #j − z)−1 χ + (L #j − z)−1 (1 − χ) = χ1 (L j − z)−1 χ − (L #j − z)−1 [Q #j , χ1 ](L j − z)−1 χ + (1 − χ0 )(Q #j − z)−1 (1 − χ) + (L #j − z)−1 [Q #j , χ0 ](Q #j − z)−1 (1 − χ ). Then (L #j − z)−1 1 + [Q #j , χ1 ](L j − z)−1 χ − [Q #j , χ0 ](Q #j − z)−1 (1 − χ)




= χ1 (L j − z)−1 χ + (1 − χ0 )(Q #j − z)−1 (1 − χ ). According to assumptions (2.5) and (2.10), there exists C # such that spectrums of L j , Q #j , j = 1, 2, do not intersect the interval ]− ∞, −C # ]; hence for z ∈ ]− ∞, −C # ], the resolvents (L j − z)−1 , (Q #j − z)−1 are bounded, and we immediately obtain [Q #j , χ1 ](L j − z)−1 χ − [Q #j , χ0 ](Q #j − z)−1 (1 − χ) = O (h). Consequently, for h small enough and z ∈ ]− ∞, −C # ], the resolvent (L #j − z)−1 is bounded and z ∈ / sp(L #j ). In what follows we use the notation
N (L #j , λ) = N L #j , ]− C # , λ] , j = 1, 2. The spectral shift function ξ(λ, h) is determined modulo a constant, and from (2.10) we deduce that ξ(λ, h) is constant on ]− ∞, −C1 ] for C1 sufficiently large. In the following, without loss of the generality, we may choose ξ(λ, h) so that ξ(λ, h) = 0 on ]− ∞, −C # ]. Moreover, in this section we consider ξ(λ, h) = lim↓0 ξ(λ + , h) as a function continuous from the right. The main result in this section is a Weyl-type asymptotics for the spectral shift function. THEOREM 2 Assume that L j , j = 1, 2, satisfy the assumptions of Section 2. Let 0 < E 0 < E 1 , and suppose that each λ ∈ [E 0 , E 1 ] is a noncritical energy level for Q j , Q #j , j = 1, 2. Assume that there exist positive constants B, 1 , C1 , h 1 such that for any λ ∈ [E 0 − 1 , E 1 + 1 ], h/B ≤ δ ≤ B, and h ∈ ]0, h 1 ], we have
# N L # , [λ − δ, λ + δ] ≤ C1 δh −n , j = 1, 2. (5.1)

Then there exist ω(λ) ∈ C 1 (R), h 0 > 0 such that  2 # ξ(λ, h) = N (L #j , λ]) j=1 + ω(λ)h −n + O (h 1−n ) uniformly with respect to λ ∈ [E 0 , E 1 ] and h ∈ ]0, h 0 ].

(5.2)

406

BRUNEAU and PETKOV

Remark. Notice that if λ is a noncritical energy level, then for  > 0 small enough each µ ∈ ]λ − , λ + [ is also a noncritical one. Consequently, (5.2) remains valid on some interval [E 0 − α, E 1 + α], α > 0. Recall that the operators L #j , j = 1, 2, have been defined in Section 2 by using the operators Q #j , j = 1, 2, whose coefficients satisfy a #j,ν (x; h) = a j,ν (x; h) for |x| ≤ R, R > R0 . If the principal symbol l j (x · ξ ) of Q j is noncritical for λ ∈ [E 0 , E 1 ], we can extend a #j,ν (x; h) for |x| > R in such a way that every λ ∈ [E 0 , E 1 ] becomes noncritical for Q #j . This continuation changes the operator L #j , but as has been proved by Bony [1], assumption (5.1) does not depend on the continuation of a #j,ν (x; h). To prove Theorem 2, we introduce an intermediate operator exploiting the following result of Bony (see also [27]). 3 (see [2, Sec. 2.3]) Assume that L satisfies the assumptions of Section 2, and suppose that each λ ∈ [E 0 , E 1 ] is a noncritical energy level for Q. Given a fixed λ ∈ [E 0 , E 1 ], there exists a differential operator L˜ such that ˜ satisfies the assumptions of Section 2 with n = n + 1; (a) the pair (L , L) (b) there exists an interval I0 3 λ, such that each µ ∈ I0 is a nontrapping and ˜ noncritical energy level for L; ˜ (c) the operator L has no resonances in a complex neighborhood 0 of I0 , and 0 is independent on h. PROPOSITION

Now denote by ξ(λ; A, B) the spectral shift function related to the operators A and B. Using Proposition 3 for the operator L 1 , we can construct an operator L˜ 1 and decompose the spectral shift function ξ(λ; L 1 , L 2 ) as follows: ξ(λ; L 1 , L 2 ) = ξ(λ; L 1 , L˜ 1 ) − ξ(λ; L 2 , L˜ 1 ). Here L 2 , L˜ 1 satisfy the assumptions of Section 2 since we may estimate the difference L 2 − L˜ 1 = (L 2 − L 1 ) + (L 1 − L˜ 1 ) by applying our assumptions on Q 1 − Q 2 . Thus it is sufficient to prove Theorem 2 for λ ∈ I2 ⊂ I0 and the pair (L 1 , L 2 ) with L 2 = Q 2 being a differential operator having no resonances in a complex neighborhood 0 of I0 and such that every λ ∈ I0 is a nontrapping and noncritical energy level for L 2 . Then the assertion follows by applying the local result and covering the compact interval [E 0 , E 1 ] by small intervals. We denote by ξ(λ, h) the spectral shift function for the operators (L 1 , L 2 ). Applying Theorem 1 in the domain 0 , we deduce that there exists a function g+ (z, h)

MEROMORPHIC CONTINUATION

407

holomorphic in 0 such that for λ ∈ I0 = W0 ∩ R, W0 b 0 , we have ξ 0 (λ, h) =

1 Im g+ (λ, h) + π

X w∈Res L 1 ∩0 Im w6=0

− Im w + π|λ − w|2

X

δ(λ − w), (5.3)

w∈Res L 1 ∩I0

where g+ (z, h) satisfies the estimate #

|g+ (z, h)| ≤ C(W0 )h −n ,

z ∈ W0 ,

(5.4)

with C(W0 ) > 0 independent on h ∈ ]0, h 0 ]. In the following, we fix an open interval I0 ⊂ R+ so that each µ ∈ I0 is a noncritical energy level for Q j , j = 1, 2, and we introduce open intervals I2 b I1 b I0 . It is convenient to decompose ξ(λ, h) for λ ∈ I2 into a sum of a term independent on λ and a second one localized in I0 , where (5.3) holds. LEMMA 2 Let C # > 0 be such that the spectrums of L j and L #j , j = 1, 2, do not intersect the interval [−∞, −C # ]. Let ϕ1 , ϕ2 ∈ C0∞ (R; R+ ) be such that supp ϕ1 ⊂ (−∞, γ1 ), supp ϕ2 ⊂ I1 , ϕ2 = 1 on I2 = (γ1 , γ2 ), and ϕ1 + ϕ2 = 1 on [−C # − η0 , γ2 ], η0 > 0. Then for λ ∈ I2 we have  2 ξ(λ, h) = trbb ϕ1 (L j ) j=1 + G ϕ2 (λ) + Mϕ2 (λ), (5.5)

where G ϕ2 (λ) = Mϕ2 (λ) =

1 π

Z ]−∞,λ]

X

Im g+ (µ, h)ϕ2 (µ) dµ, Z − Im w ϕ (µ) dµ + 2 2 ]−∞,λ] π|µ − w|

w∈Res L 1 ∩0 Im w6=0

X w∈Res L 1

ϕ2 (w),

∩ ]−C # ,λ]

(5.6) and we omit in Mϕ2 and G ϕ2 the dependence of h. Proof Roughly speaking, for λ ∈ I2 , if we express the action of the distributions as integrals, we must have Z λ Z λ ξ(λ, h) = ϕ1 (µ)ξ 0 (µ, h) dµ + ϕ2 (µ)ξ 0 (µ, h) dµ. −∞

−∞

Since ϕ1 vanishes on I2 , the first term is independent on λ ∈ I2 and equal to trbb [ϕ1 (L j )]2j=1 . For the second one, we may apply (5.3) since ϕ2 is supported in I1 ⊂ I0 .

408

BRUNEAU and PETKOV

For a rigorous proof of the above representation, take f ∈ C0∞ (I2 ) and introduce F(λ) = (ϕ1 + ϕ2 )(λ)

+∞

Z

f (µ) dµ,

λ

which is compactly supported. Since supp f ⊂ I2 and ϕ1 + ϕ2 = 1 on I2 , we have Z +∞ 0 0 0 F (λ) = − f (λ) + (ϕ1 + ϕ 2 )(λ) f (µ) dµ, λ

where the second term vanishes on [−C # − η0 , +∞[. Our choice of ξ(λ, h) = 0 on ]− ∞, −C # ] makes it possible to write hξ, f iD 0 ,D = −hξ, F 0 iD 0 ,D = hξ 0 , FiD 0 ,D . R +∞ R Next, the equality ϕ1 λ f = ϕ1 R f yields Z D 0 ξ , ϕ1

+∞ λ

f

E D 0 ,D

=

Z



f hξ , ϕ1 iD 0 ,D = 0

R

Z R

  2 f trbb ϕ1 (L j ) j=1 .

For the term involving ϕ2 , we apply (5.3) and we get Z +∞ E Z +∞ E Z D D D 0 0 0 ξ , ϕ2 f 0 = G ϕ2 , ψ f 0 + M ϕ2 , ψ λ

D ,D

λ

D ,D

λ

+∞

f

E D 0 ,D

for ψ ∈ C ∞ (R) equal to 1 on R+ and vanishing on ]− ∞, −1]. The above relations imply (5.5) in the sense of distributions since G ϕ2 ψ 0 = Mϕ2 ψ 0 = 0 and ψ f = f . To prove Theorem 2, we apply a Tauberian argument for the increasing function Mϕ2 (λ). Consider a function θ(t) ∈ C0∞ (] − δ1 , δ1 [), θ(0) = 1, θ (−t) = θ (t), so ˆ that the Fourier transform θˆ of θ satisfies θ(λ) ≥ 0 on R, and assume that there exist ˆ 0 < 0 < 1, δ0 > 0, so that θ(λ) ≥ δ0 > 0 for |λ| ≤ 0 . Next, introduce Z −1 −1 (Fh θ)(λ) = (2π h) eitλ/ h θ(t) dt = (2π h)−1 θˆ (−h −1 λ). Remark. It is obvious that Lemma 2 holds if we take a partition of unity ϕ12 + ϕ22 over [−C # − η0 , γ2 ] with cut-off functions ϕ j , j = 1, 2. The next lemma permits us to establish a connection between the asymptotics of the functions Mϕ2 and Nϕ#2 . LEMMA 3 Let ϕ2 ∈ C0∞ (I1 ; R+ ), and let Nϕ#2 (λ) = tr(ϕ2 (L #1 )1]−C # ,λ] (L #1 )). Then there exists

MEROMORPHIC CONTINUATION

409

ωϕ2 (λ) ∈ C00 (I0 ) such that for any λ ∈ R we have d d # (F −1 θ ∗ Mϕ2 )(λ) = (F −1 θ ∗ Nϕ#2 )(λ) − G 0ϕ2 (λ) + ωϕ2 (λ)h −n + O (h 1−n ), dλ h dλ h (5.7) # where O (h 1−n ) is uniform with respect to λ ∈ R. Moreover, we have #

Mϕ2 (λ) = (Fh−1 θ ∗ Mϕ2 )(λ) + O (h 1−n ) =

(Fh−1 θ

∗

Nϕ#2 )(λ) −

G ϕ2 (λ) +

Z

λ

−∞

#

ωϕ2 (µ) dµh −n + O (h 1−n ) (5.8)

uniformly with respect to λ ∈ I0 . Proof For simplicity of notation, we omit the subscript ϕ2 and denote by M, G, N # , ω the functions Mϕ2 , G ϕ2 , Nϕ#2 , ωϕ2 . According to (5.6) and (5.3), for any λ ∈ R we have d (F −1 θ ∗ M)(λ) = (Fh−1 θ ∗ M 0 )(λ) = (Fh−1 θ ∗ ϕ2 ξ 0 )(λ) − (Fh−1 θ ∗ G 0 )(λ). dλ h #

Using the Cauchy inequalities, it easily follows that G 0 (λ) = O (h −n ) and G 00 (λ) = # O (h −n ), and we immediately obtain d # (Fh−1 θ ∗ G)(λ) = G 0 (λ) + O (h 1−n ) dλ uniformly with respect to λ ∈ R. It remains to examine (Fh−1 θ

1 ∗ ϕ2 ξ )(λ) = 2π h 0

Z

 2 −1 −1 eitλh θ(t) trbb e−ith L j ϕ2 (L j ) j=1 dt.

We prove that (Fh−1 θ ∗ ϕ2 ξ 0 )(λ) =

d (F −1 θ ∗ N # )(λ) + ω(λ)h −n + O (h 1−n ), dλ h

λ ∈ R, (5.9)

where ω(λ) ∈ C00 (I0 ) has compact support and O (h 1−n ) is uniform with respect to ˜ n with R˜ > λ ∈ R. As in Section 2, define the operator L #1 on the torus TR˜ = (R/ RZ) ∞ ˜ 2R > 2R0 , and introduce χ ∈ C0 ({x : |x| ≤ R}) equal to 1 for |x| ≤ 2R > 2R0 . We have  2 
2 −1 −1 trbb e−ith L j ϕ2 (L j ) j=1 = tr χ e−ith L j ϕ2 (L j )χ j=1  2 −1 + trbb e−ith L j ϕ2 (L j )(1 − χ 2 ) j=1 .

410

BRUNEAU and PETKOV

Applying the Duhamel formula and the semiclassical Egorov theorem (see [6, Sec. 6] for more details), for |t| sufficiently small we obtain  2  2 −1 −1 trbb e−ith L j ϕ2 (L j )(1 − χ 2 ) j=1 = tr e−ith Q j ϕ2 (Q j )(1 − χ 2 ) j=1 + O (h ∞ ), tr χ e−ith

−1 L

1



−1 # ϕ2 (L 1 )χ = tr χ e−ith L 1 ϕ2 (L #1 )χ + O (h ∞ )
−1 # = tr e−ith L 1 ϕ2 (L #1 )
−1 # − tr e−ith Q 1 ϕ2 (Q #1 )(1 − χ 2 ) + O (h ∞ ),

where Q #1 is a differential operator Q #1 =

X

# a1,ν (x; h)(h D)ν

|ν|≤2 # (x; h) = a (x; h) for |x| < r , on the torus TR˜ introduced in Section 2 and a1,ν 1,ν 0 r0 > 2R0 . Using the classical constructions of a parametrix for small |t| for the −1 # −1 unitary groups e−ith Q 1 , e−ith L 2 , combined with the fact that λ ∈ I0 is noncritical for Q #1 , L 2 , we deduce for λ ∈ I0 ,
tr (Fh−1 θ)(λ − Q #1 )ϕ2 (Q #1 )(1 − χ 2 ) = ω1 (λ)h −n + O (h 1−n ),
tr χ (Fh−1 θ)(λ − L 2 )ϕ2 (L 2 )χ = ω2 (λ)h −n + O (h 1−n ),

with functions ω1 , ω2 ∈ C00 (I1 ) and O (h 1−n ) uniform with respect to λ ∈ I0 . The problem can be reduced to the application of the stationary phase method to some integrals where the integration is over a compact set. We refer to [10, Chap. 10] for more details. Since θˆ ∈ S (R), we can extend the above relations to all λ ∈ R with O (h 1−n ) uniform with respect to λ ∈ R. For the trace involving Q j , j = 1, 2, we have for λ ∈ I0 ,  2 tr (Fh−1 θ)(λ − Q j )ϕ2 (Q j )(1 − χ 2 ) j=1 = ωext (λ)h −n + O (h 1−n ) (5.10) with ωext ∈ C00 (I0 ) and O (h 1−n ) uniform with respect to λ ∈ I0 . The proof of (5.10) is more technical since we must integrate over a noncompact domain. In fact, it is similar to the calculation of the traces in [2, Sec. 4], and for the sake of completeness we present a proof in the appendix. Moreover, we show in the appendix that we can extend (5.10) to all λ ∈ R with O (h 1−n ) uniform with respect to λ ∈ R. Taking together the asymptotics of the traces and the above relations, we obtain (5.9) and (5.7). Now we apply a Tauberian theorem (see, e.g., [24, Th. V-13]) for the increasing function Mϕ2 (λ). For this purpose, we need the estimates #

Mϕ2 (λ) = O (h −n ),

d # (Fh−1 θ ∗ Mϕ2 )(λ) = O (h −n ), dλ

∀λ ∈ R.

(5.11)

MEROMORPHIC CONTINUATION

411

The first one follows easily from (5.6). To establish the second one, we apply equality (5.7). Thus it suffices to prove the estimate   #  L − λ d # (Fh−1 θ ∗ Nϕ#2 )(λ) = (2π h)−1 tr θˆ 1 ϕ2 (L #1 ) = O (h −n ), dλ h

∀λ ∈ R.

(5.12) To do this, assume first that λ ∈ [E 0 − 1 , E 1 + 1 ]. Taking into account (5.1), we obtain   #  µ − λ X L − λ ϕ2 (L #1 ) = θˆ ϕ2 (µ) tr θˆ 1 h h # µ∈sp(L 1 )∩supp ϕ2

≤

C/ Xh

X

θˆ

k=0 kh/B≤|µ−λ|≤(k+1)h/B



≤C h

1−n #

+

C/ Xh k=1

µ − λ ϕ2 (µ) h

# (k + 1)h 1−n  # ≤ Ch 1−n , 3 k

(5.13)

ˆ where we have used the inequality |θ(µ)| ≤ C(1 + |µ|)−3 . On the other hand, for λ∈ / [E 0 − 1 , E 1 + 1 ] and µ ∈ supp ϕ2 , we have |µ − λ| ≥ δ2 > 0, and the term (5.11) is estimated by O (h ∞ ). Now a Tauberian argument implies the first assertion in (5.8). The second one is obtained by integration of (5.7) over [inf I0 , λ] combined with the equalities Mϕ2 (µ) = G ϕ2 (µ) = Nϕ#2 (µ) = 0,

µ ≤ inf I1 ,

ˆ ∈ S (R). and the fact that θ(t) Proof of Theorem 2 As it was mentioned above, it remains to show that #

ξ(λ, h) = ξ(λ; L 1 , L 2 ) = N (L #1 , λ) + ω0 (λ)h −n + O (h 1−n ),

λ ∈ I2 ,

(5.14)

for a differential operator L 2 = Q 2 having no resonances in 0 and such that each λ ∈ I0 is a nontrapping and noncritical energy level for L 2 . According to Lemmas 2 and 3, for λ ∈ I2 we have Z λ  2 # −1 # ξ(λ, h) = trbb ϕ1 (L j ) j=1 + (Fh θ ∗ Nϕ2 )(λ) + ωϕ2 (µ) dµh −n + O (h 1−n ). −∞

Given a function χ ∈ C0∞ (Rn ), χ = 1 on B(0, R0 ), exploiting the functional calculus

412

BRUNEAU and PETKOV

for smooth functions and the estimates for the trace (see [29]), we obtain  2 
2  2 trbb ϕ1 (L j ) j=1 = tr χϕ1 (L j )χ j=1 + trbb ϕ1 (L j )(1 − χ 2 ) j=1

= tr χϕ1 (L #1 )χ − tr χϕ1 (L 2 )χ  2 + tr ϕ1 (Q j )(1 − χ 2 ) j=1 + O (h ∞ )
= tr ϕ1 (L #1 ) + C(ϕ1 )h −n + O (h 1−n ), where C(ϕ1 ) is a constant depending on ϕ1 . # On the other hand, applying a Tauberian theorem for Nϕ#2 (λ) = O (h −n ), we deduce # Nϕ#2 (λ) = (Fh−1 θ ∗ Nϕ#2 )(λ) + O (h 1−n ), ∀λ ∈ R. Consequently, for λ ∈ I2 we get

ξ(λ, h) = tr ϕ1 (L #1 ) + tr ϕ2 (L #1 )1]−C # ,λ] (L #1 ) Z λ   # + C(ϕ1 ) + ωϕ2 (µ) dµ h −n + O (h 1−n ). −∞

By construction, we have ϕ1 (L #j ) + ϕ2 (L #j )1]−C # ,λ] (L #j ) = 1]−C # ,λ] (L #j ),

∀λ ∈ I2 ,

Rλ and this implies (5.14) with ω0 (λ) = C(ϕ1 ) + −∞ ωϕ2 (µ) dµ ∈ C 1 (R). SF To obtain (5.2), we construct a covering of the interval [E 0 , E 1 ] ⊂ ν=1 Jν by small open intervals Jν , so that for every Jν we can find an operator Q ν with the properties of Proposition 3, where I0 is replaced by Jν . Next, we introduce a partition of unity F X ϕν (x) = 1 on [E 0 , E 1 ], ϕν ∈ C0∞ (Jν ; R+ ), ν=1

and we apply the above argument. This completes the proof of Theorem 2. 6. Breit-Wigner approximation In this section we consider small domains of width h, and we prove a semiclassical analogue of the Breit-Wigner approximation for ξ(λ, h) (see [22], [23], [4] for similar results, [13] for the case of a potential having the form of a “well in the island,” and [12] for a one-dimensional critical case). In the following, η(λ, h) denotes the real analytic function defined by  2 η(λ, h) = ξ(λ, h) − #{µ ∈ [E 0 , λ] : µ ∈ sppp (L j (h))} j=1 .

MEROMORPHIC CONTINUATION

413

THEOREM 3 Assume that L j (h), j = 1, 2, satisfy the assumptions of Theorem 2. Then for any λ ∈ [E 0 , E 1 ], any 0 < δ < h/B, 0 < B1 < B, and h sufficiently small, we have

η(λ+δ, h)−η(λ−δ, h) =

h

i2 ωC− (w, [λ−δ, λ+δ])

X

j=1

w∈Res L j (h) Im w6=0, |w−λ|
#

+O (δ)h −n , (6.1)

where B > 0 is the constant introduced in Theorem 2. Remark. Following the result of Bony [1], assumption (5.1) implies the existence of positive constants D, C3 , h 3 such that for λ ∈ [E 0 , E 1 ], h/D ≤ δ ≤ D, and h ∈ ]0, h 3 ] we have  # # z ∈ C : z ∈ Res L(h), |z − λ| ≤ δ ≤ C3 δh −n . (6.2) Proof We apply Theorem 1 in the interval I0 ⊃ (λ − δ, λ + δ), 0 < δ ≤ h/B1 , and we introduce the function h  1 X 1 i2 F(z, h) = − , z ∈ D(λ, h/B). z−w z − w j=1 w∈Res L j (h), Im w6=0 h/B1 ≤|w−λ|≤C4

It is sufficient to show that #

|F(z, h)| ≤ Ch −n ,

|z − λ| ≤ h/B.

(6.3)

We have ∂z F(z, h) =

h

X w∈Res L j (h), Im w6=0 h/B1 ≤|w−λ|≤C4

i2 1 1 − . (z − w)2 (z − w)2 j=1

Let l0 ∈ N be an integer such that D ≤ 2l0 −1 B. Following the argument in [23] and

414

BRUNEAU and PETKOV

applying (6.2), for any z ∈ D(λ, h/B) we obtain X X 1 ≤ |z − w|2 w∈Res L j (h), Im w6=0 h/B1 ≤|w−λ|
w∈Res L j (h), Im w6=0 h/B1 ≤|w−λ|≤2l0 h/D

1 |z − w|2

C log(1/ h)

X

X

k=l0

2k h/D≤|w−λ|≤2k+1 h/D

+

C log(1/ h) l0

≤ C2 D

−1 −1−n #

h

+C

X k=l0

≤ Ch

−1−n #

1 |z − w|2

(2k+1 h)h −n (2k h)2

#

.

Here and below we denote by C > 0 the different constants that may change from line to line and that are independent on h and the choice of λ in the interval [E 0 , E 1 ]. Thus we get the estimate |∂z F(z, h)| ≤ Ch −n

# −1

,

z ∈ D(λ, h/B).

It remains to find an estimate of |F(µ0 , h)| = | Im F(µ0 , h)| at a suitable point µ0 = µ0 (h).∗ Set ν = h/B < h/B1 , and suppose that for all µ ∈ R, |µ − λ| ≤ ν, we have # | Im F(µ, h)| ≥ Mh −n , M > 0. The continuity of the function Im F(µ, h) implies that Im F(µ, h) is either positive or negative in [λ − ν, λ + ν]. Assuming Im F(µ, h) positive, we get Z λ+ν # Mh −n +1 1 ≤ Im F(µ, h) dµ Bπ 2π λ−ν Z X 1 λ+ν h | Im w| i2 ≤ dµ π λ−ν |µ − w|2 j=1 w∈Res L j (h), Im w6=0 |w−λ|≤C

2 Z 1 X λ+ν + π λ−ν j=1

X w∈Res L j (h), Im w6=0 |w−λ|
| Im w| dµ |µ − w|2

# ≤ η(λ + ν, h) − η(λ − ν, h) + Ch 1−n . Here we have used the inequality Z ∞ Z λ+ν | Im w| | Im w| dµ ≤ dµ ≤ π 2 |µ − w| |µ − w|2 λ−ν −∞ ∗ There is some similarity between the proof of the existence of µ

0 (h) and that of the existence of a suitable point # z 0 (h), Im z 0 (h) ≥ δ > 0, in [23, Sec. 4], so that log | det S(z 0 (h), h)| ≥ −Ch −n .

MEROMORPHIC CONTINUATION

415

and (6.2) to estimate the number of resonances in {w : |w − λ| < h/B1 }. Notice that if D ≤ B1 , we have {w : |w − λ| < h/B1 } ⊂ {w : |w − λ| < h/D}. Next, assumption (5.1) combined with Theorem 2 yields the estimate ξ(λ + ν, h) − ξ(λ − ν, h) ≤ Ch 1−n # . Thus η(λ + ν, h) − η(λ − ν, h) ≤ ξ(λ + ν, h) − ξ(λ − ν, h) +

2 X  # ] µ ∈ sppp (L j ) : |µ − λ| ≤ ν ≤ Ch 1−n , j=1

where for the second inequality we have once more used (6.2), observing that the positive eigenvalues of L j coincide with the resonances on R+ . Consequently, we obtain a bound for M. Hence there exist a constant C > 0 and µ0 ∈ [λ − ν, λ + ν], so that # |F(µ0 , h)| ≤ Ch −n . (6.4) Writing F(z, h) = F(µ0 , h) +

z

Z

µ0

∂z F(z, h) dz, |z − λ| ≤ h/B,

we obtain (6.3). The case of Im F(µ, h) < 0 can be treated by the same argument, # exploiting the inequality − Im F(µ, h) ≥ Mh −n , |µ − λ| ≤ ν. By an integration over the interval (λ − δ, λ + δ), we complete the proof of (6.1). Remark. Our proof goes without a factorization in small domains {z ∈ C : |z − λ| ≤ Ch} and a suitable trace formula (see [23, Lem. 6.2] and [4, Th. 1.3]). The above argument can be applied to simplify the proof of [23, Lem. 6.2]. Next, estimate (6.3) of F(z, h) immediately yields the following. COROLLARY 1 Under the assumptions of Theorem 3 for µ ∈ R, |µ − λ| < h/B, we have the representation

ξ 0 (µ, h) =

1 Im q(µ, h) π h X + w∈Res L j (h), |w−λ|
− Im w + π|µ − w|2

X w∈(Res L j (h)∩R) |w−λ|
i2 δ(µ − w)

j=1

,

(6.5)

416

BRUNEAU and PETKOV

where q(z, h) = p(z, h)− p(z, h), p(z, h) is holomorphic in D(λ, h/B), and p(z, h) satisfies the estimate #

| p(z, h)| ≤ Ch −n ,

z ∈ D(λ, h/B),

with C > 0 independent on h ∈ ]0, h 0 ] and λ ∈ [E 0 , E 1 ]. We may slightly improve Theorem 3, noting that for every 0 <  < 1 and |µ − λ| ≤ h/B we have X w∈Res L j (h) h/B1 ≤|w−λ|≤h/B1

| Im w| h # # ≤ 2 2 Ch 1−n = O (h −n ). |µ − w|2  h

Thus for 0 < δ ≤ h/B equality (6.1) can be replaced by η(λ + δ, h) − η(λ − δ, h) =

h

i2 ωC− (w, [λ − δ, λ + δ])

X

j=1

w∈Res L j (h), Im w6=0 |w−λ|≤h/B1 #

+ O (δ)h −n .

(6.6)

To obtain a stronger version involving the resonances in smaller “boxes,” we need some additional information for the distribution of the resonances in {w ∈ C : |w − λ| ≤ h}. In the case of the Schr¨odinger operator L(h) = −h 2 1 + V (x) with V (x) ∈ C0∞ (Rn ) real valued, this is possible by applying the recent result of Stefanov [31]. Set a0 (x, ξ ) = |ξ |2 + V (x), and let 0 < E 0 < E 1 be noncritical values of a0 (x, ξ ). Let a0−1 [E 0 , E 1 ] = Wint ∪ Wext , where Wext is the unbounded connected component, while Wint is the union of bounded ones if there are such connected components. Assume that all points in Wext are nontrapping (see [31] for a precise definition). Then, according to [31, Th. 6.1], there exists a function 0 < R1 (h) = O (h ∞ ) such that for any M ∈ N the operator L(h) has no resonances in the set  M (λ, h) = [E 0 , E 1 ] + i[−Mh, −R1 (h)],

0 < h ≤ h(M).

(6.7)

√ Setting 0 < R(h) = h R1 (h) = O (h ∞ ), an elementary argument shows that for λ ∈ [E 0 , E 1 ] and |µ − λ| ≤ R(h)/2 we have X w∈Res L(h), | Im w|≤R1 (h) R(h)≤| Re w−λ|≤h

| Im w| # ≤ Ch −n . 2 |µ − w|

MEROMORPHIC CONTINUATION

417

In the next result we treat a formally symmetric differential operator X L 1 (h) = aα (x, h)(h Dx )α |α|≤2

on L 2 (Rn ) satisfying the assumptions of Section 2. Given a fixed λ ∈ ]E 0 , E 1 [ , as in Section 5, we may construct an operator L 2 (h) having properties (a) – (c) of Proposition 3. Applying Theorem 3 for L j (h), j = 1, 2, and {z ∈ C : |z − λ| ≤ h/B1 } ⊂ W , and assuming that we have a free resonance domain, we obtain the following improvement of Corollary 1. 2 Let E 0 < λ < E 1 be fixed. Let L 2 (h) be chosen so that L j (h), j = 1, 2, satisfy the assumptions of Theorem 3; L 2 (h) has no resonances in the disk {z ∈ C : |z − λ| ≤ h/B1 }. Suppose that there exists a function 0 < R1 (h) = O (h ∞ ) such that L 1 (h) has no resonances in the set COROLLARY

[E 0 , E 1 ] + i[−h, −R1 (h)],

 > 0, 0 < h ≤ h().

Then for |µ − λ| < R(h)/2 and h sufficiently small we have ξ 0 (µ, h) =

1 Im q(µ, h) + π +

X

X w∈Res L 1 (h), | Re w−λ|
− Im w π|µ − w|2

δ(µ − w)

(6.8)

w∈Res L 1 (h)∩R |w−λ|
with R(h) =

√

h R1 (h) = O (h ∞ ) and q(µ, h) as in Corollary 1.

7. Local trace formula In this section we prove a local trace formula that is a slightly stronger version of that in [28] and [29] (see [23] for compactly supported perturbations). Exploiting Theorem 1, we repeat with trivial modifications the argument of [23, Sec. 5] to get the following. THEOREM 4 Assume that L j (h) satisfy the assumptions of Section 2. Let  b ei]−2θ0 ,2θ0 [ ]0, ∞[ be an open, simply connected, relatively compact set such that I =  ∩ R is an interval. Suppose that f is holomorphic on a neighborhood of  and that ψ ∈ C0∞ (R) satisfies ( 0, d(I, λ) > 2, ψ(λ) = 1, d(I, λ) < ,

418

BRUNEAU and PETKOV

where  > 0 is sufficiently small. Then h  2 trbb (ψ f )(L j (h)) j=1 =

i2 f (z)

X

j=1

z∈Res L j (h)∩

+ E , f,ψ (h)

(7.1)

with  # |E , f,ψ (h)| ≤ M(ψ, ) sup | f (z)| : 0 ≤ d(, z) ≤ 2, Im z ≤ 0 h −n . Proof Choose an almost analytic extension ψ˜ of ψ so that ψ˜ ∈ C0∞ (C), ψ˜ = 1 on , and  supp ∂ z ψ˜ ⊂ z ∈ C :  ≤ d(, z) ≤ 2 . Setting  = {z ∈ C : d(, z) ≤ }, we have  2 trbb (ψ f )(L j (h)) j=1 = hξ 0 (λ, h), (ψ f )(λ)i h i2 X = (ψ f )(w)

1 + j=1 2πi

w∈Res L j (h)∩supp ψ

+

1 2πi

Z

h (ψ f )(λ)



X w∈Res L j (h)∩2 Im w6=0

Z

(ψ f )(λ)r (λ, h) dλ

1 1 i2 − dλ. λ − w λ − w j=1

The integral involving r (λ, h) can be estimated using (1.4) with W = 2 . For the integral containing the resonances, we apply the Green formula, and we get the term h

i2 (ψ˜ f )(z)

X

z∈Res L j (h), Im z6=0

+

1 π

Z

j=1

h ˜ (∂ z ψ)(z) f (z) C−

X

w∈Res L j (h)∩2 Im w6=0



1 1 i2 − L (dz), z−w z − w j=1

where L (dz) is the Lebesgue measure on C. As in the proof of [23, Th. 1], we apply the inequality Z p 1 L (dz) ≤ 2 2π|1 | 1 |z − w| and an upper bound for the number of the resonances in 2 to obtain the result. Since we have no restrictions on the behavior of the holomorphic function f (z) on  ∩ {Im z > 0}, we may apply the above argument choosing f (z) = e−it z/ h , t ∈ R, to get the following.

MEROMORPHIC CONTINUATION

419

THEOREM 5 Let  and ψ be as in Theorem 4, and let ψ˜ ∈ C0∞ (C) be an almost analytic extension of ψ supported in 2 . Then for any 0 < δ < 1 and t ≥ h δ , we have h i2 X  2 −itw/ h ˜ trbb ψ(L j (h))e−i(t/ h)L j (h) j=1 = ψ(w)e + Oδ (h ∞ ). j=1

w∈Res L j (h)∩2

(7.2) > 0 such that for 0 < h ≤ h N

Moreover, for t ≥  > 0 and N ∈ N there exists h N we have i2 h X  2 −itw/ h ˜ ψ(w)e trbb ψ(L j (h))e−i(t/ h)L j (h) j=1 =

j=1

w∈Res L j (h)∩2 | Im w|≤−N h log h

+ O (h N −n ). #

(7.3) Proof Choose an almost analytic extension ψ˜ of ψ as in Theorem 4. Applying the Green formula, we must examine the integrals Z −it z/ h ˜ ∂ z ψ(z)e r (z, h)L (dz), C−

Z C−

−it z/ h ˜ ∂ z ψ(z)e

h

X

w∈Res L j (h)∩2



1 1 i2 − L (dz). z−w z − w j=1

Choose µ > 0, 0 < δ + µ < 1. For −h µ ≤ Im z ≤ 0, we have ˜ ≤ C N | Im z| N ≤ C N h µN , |∂ z ψ|

∀N ∈ N,

and the integration over −h µ ≤ Im z ≤ 0 combined with the argument of the proof of Theorem 4 yields a term bounded by O (h ∞ ). On the other hand, for t ≥ h δ , Im z ≤ −h µ , we get µ−1 δ+µ−1 |e−it z/ h | ≤ e−th ≤ e−h = Oδ (h ∞ ), and this implies (7.2). For the second assertion, we have |e−itw/ h | ≤ et N log h ≤ h N  for | Im w| ≥ −N h log h, and this completes the proof. Remark. For nontrapping compactly supported perturbations L(h) (see [32], [7]) and for nontrapping long-range perturbations L(h) = −h 2 1 + V (x) of the Laplacian (see [16]), there are no resonances of L(h) in the domain −N h log

1 ≤ Im z ≤ 0, h

0 < h ≤ hN .

For such perturbations, the right-hand side of (7.3) is equal to O (h N −n ), and we obtain an analogue of the classical trace formula for nontrapping perturbations. #

420

BRUNEAU and PETKOV

8. Existence of resonances close to the real axis In this section we consider the operator L(h) = −h 2 1g + V (x), where 1g is the symmetric Laplace-Beltrami operator on L 2 (Rn ) associated to a metric g(x) = {gi, j (x)}1≤i, j≤n and V (x) ∈ C ∞ (Rn ) is a real-valued function. We assume that there exists ρ > n so that α
∂ gi, j (x) − δi, j + |∂ α V (x)| ≤ Cα hxi−ρ−|α| , 1 ≤ i, j ≤ n, ∀α. (8.1) x x Moreover, we assume that the coefficients {gi, j (x)} and V (x) can be extended holomorphically in x to the domain given in (2.8), and we assume that estimate (8.1) holds in this domain. Consider the symbol a0 (x, ξ ) = hg(x)−1 ξ, ξ i + V (x), denote by Ha0 the Hamilton vector field associated to a0 , and denote by 8t = exp(t Ha0 ) the Hamilton flow. Given λ > 0, let 6λ = {(x, ξ ) ∈ Rn : a0 (x, ξ ) = λ} be the energy surface, and let ∇a0 (x, ξ ) 6= 0 on 6λ . A point ν ∈ 6λ is called periodic if there exists T > 0 such that 8T (ν) = ν, and the smallest T > 0 with this property is called period T (ν) of ν. Given a periodic point ν, consider the trajectory   γ (ν) = 8t (ν) : 0 ≤ t ≤ T (ν) = (x(t), ξ(t)) : 0 ≤ t ≤ T (ν) , and define the action S(ν) along γ (ν) by Z Z S(ν) = ξ dx = γ (ν)

T (ν)

ξ(t)x 0 (t) dt.

0

Next, we denote by m(ν) ∈ Z4 the Maslov index related to γ (ν) and set q(ν) = −(π/2)m(ν). Let 5 be the set of all periodic points on 6λ , and let Z   Q(h, r ) = (2π)−n π − h −1 S(ν) + q(ν) − r T (ν) 2π T (ν)−1 dν, (8.2) 5

where dν is the Liouville measure on 6λ and the residue −π < [z]2π ≤ π is defined so that z = [z]2π + 2π k, k ∈ Z. The set 5 is bounded, the integrand in (8.2) is a measurable function, and T (ν) ≥ T0 > 0, ∀ν ∈ 5. The oscillatory function Q(h, r ) has been introduced in [20] for the analysis of the semiclassical behavior of the eigenvalues, and it is a semiclassical analogue of the oscillating function defined by T. Gureev and Yu. Safarov [15] and Safarov [26]. Notice that the limits Q(h, r ± 0) = lim↓0 Q(h, r ± ) exist for each r and 0 < h ≤ h 0 , and, moreover, Z dν Q(h, r + 0) − Q(h, r − 0) = (2π )1−n , T (ν) h,r where h,r = {ν ∈ 5 : h −1 S(ν) − q(ν) + r T (ν) ≡ 0(2π)}. Following the arguments in [22, Sec. 6], we prove the following.

MEROMORPHIC CONTINUATION

421

THEOREM 6 Let L(h) = −h 2 1g + V (x), where the metric g(x) and V (x) satisfy estimate (8.1), and let ∇a0 (x, ξ ) 6= 0 on 6λ , λ > 0. Assume that there exist an integer p ∈ Z and a subset 50 ⊂ 5 with positive Liouville measure µ(50 ) > 0, so that
  q(ν) − h −1 S(ν) 2π + 2π p T (ν)−1 = r (h), 0 < h ≤ h 0 ,

does not depend on ν ∈ 50 . Then for every 0 < η ≤ 1 and 0 < h ≤ h 1 (η) we have  (2π )1−n 1−n # w ∈ Res L(h) : |w − λ − r (h)h| ≤ ηh ≥ h 2

Z 50

dν . T (ν)

(8.3)

Remark. Clearly, |r (h)| ≤ max{|2 p − 1|, |2 p + 1|}π(T0 )−1 . Recently, Bony [3] proved that if the Liouville measure of the periodic points on 6λ is zero, then for every 0 < η ≤ 1 and for h small enough we have the upper bound  √ # w ∈ Res L(h) : |w − λ| ≤ ηh ≤ C ηh 1−n with a constant C > 0 independent on η and h. Proof Consider the scattering phase σ (λ, h) = (1/(2πi)) det S(λ, h), where the scattering operator S(λ, h) is related to L(h) and L 0 (h) = −h 2 1. According to the BirmanKrein theory (see, for instance, [33]), the scattering phase can be identified with the spectral shift function, and, under our assumptions, we have not embedded positive eigenvalues. Following [5, Th. 2.1], and taking |r (h)| ≤ r0 , 0 <  ≤ 0 , 0 < h ≤ h 0 , and λ > 0, we have

σ λ + (r (h) + )h, h − σ λ + (r (h) − )h, h h     i ≥ h 1−n Q h, r (h) + − Q h, r (h) − 2 2 + 2γ00 (λ)h 1−n − C0 h 1−n − o (h 1−n ), where γ0 (λ) = (2π)−n

Z Z Rn

a0 (x,ξ )≤λ

Z

 dξ d x,

dξ − |ξ |2 ≤λ

C0 > 0 is independent on r (h), , and h, and o (h 1−n ) means that for any fixed  > 0 we have o (h 1−n ) lim = 0. h↓0 h 1−n

422

BRUNEAU and PETKOV

On the other hand, for small 0 <  < η an application of (6.6) with δ = h yields the estimate

σ λ + (r (h) + )h, h − σ λ + (r (h) − )h, h  ≤ # w ∈ Res L(h) : |w − λ − r (h)h| ≤ ηh + Cη h 1−n ,

0 < h ≤ h 2 (η),

with Cη > 0 independent on , r (h), and h. We claim that    Q h, r (h) + − Q h, r (h) − ≥ −(2π )−n µ(5) + (2π)1−n 2 2 

Z 50

dν . (8.4) T (ν)

In fact, according to the representation of the oscillatory function Q(h, r ) (see, for instance, [26, Prop. 1]), we have     Q h, r (h) + − Q h, r (h) − 2 2 = −(2π )

−n

µ(5) + (2π )

1−n

Z 5

T −1 (ν)

X

 χh,k (ν) dν,

k∈Z

 is the characteristic function of the set where χh,k

 h,k = ν ∈ 5 : −T (ν) ≤ h −1 S(ν) − q(ν) + r (h)T (ν) − 2kπ < T (ν) . Obviously, for any ν ∈ 50 we get h −1 S(ν) − q(ν) + r (h)T (ν) + 2M(ν, h)π − 2 pπ = 0 with some M(ν, h) ∈ Z. Consequently, ν ∈ 50 =⇒

X

 χh,k (ν) ≥ 1,

k∈Z

and we obtain (8.4). Choosing  = (η) > 0 small enough, we arrange the inequality −(2π)−n µ(5) − (C0 + Cη ) + 2γ00 (λ) ≥ −

α0 4

R with α0 = (2π)1−n 50 dν/T (ν). Next, we fix η > 0 and  = (η) > 0 and choose 0 < h 1 (η) ≤ h 2 (η), so that for 0 < h ≤ h 1 (η) we have |o (h 1−n )| ≤

α0 1−n h . 4

Combining the above estimates for the difference σ (λ + (r (h) + )h, h) − σ (λ + (r (h) − )h, h), we complete the proof.

MEROMORPHIC CONTINUATION

423

Example (see [5, Sec. 7]) Let L(h) = −h 2 1 + V (x) with V (x) = 8a (x − y0 )(|x − y0 |2 + b), where a > 0, b > 0, and y0 ∈ Rn are fixed and 8a (x) ∈ C0∞ (Rn ), 8a (x) = 1 for √ |x| ≤ 2a. Let 0 <  < a/2, let |ξ0 | = λ − b, and let λ ∈ ]b, b + a 2 [ be a noncritical energy level for a0 (x, ξ ) = |ξ |2 + V (x). Therefore the set  50 = (x, ξ ) ∈ 6λ : |ξ − ξ0 |2 + |x − y0 |2 ≤  2 has a positive Liouville measure, and 50 ⊂ 5. Moreover, for every ν ∈ 50 we have π T (ν) = π, S(ν) = (λ − b)π, q(ν) = m 2 with m ∈ Z independent on ν. We may apply Theorem 6 with   r p (h) = (1/π) (π/2)m − h −1 (λ − b)π 2π + 2 p, p ∈ Z, to conclude that  # w ∈ Res L(h) : |w − λ − r p (h)h| < ηh ≥ (2π)−n µ(50 )h 1−n . On the other hand, for p 6= j and 0 < h ≤ h 0 we have   w : | Re w − λ − r p (h)h| < ηh ∩ w : | Re w − λ − r j (h)h| < ηh = ∅, and the clusters related to p 6= j produce different resonances. Choosing δ > 0 so that ]λ − δ, λ + δ[ ⊂ ]b, b + a 2 [ , one easily obtains  # w ∈ Res L(h) : |w − λ| ≤ δ ≥ αδ(2π )−n µ(50 )h −n with α > 0 independent on δ. A stronger asymptotics for the number of the resonances in [b, b + a 2 ] + i[−R(h), 0] has been obtained by Stefanov [31]. Notice that in estimate (8.3) we count only the resonances lying in clusters. A. Appendix In this appendix we present a proof of (5.10). Following the remark after Lemma 2, we assume that ϕ2 = ψ 2 , ψ ∈ C0∞ (I1 ; R+ ), I1 ⊂ I0 . Recall that λ ∈ I0 , supp θ(t) ⊂ [−δ1 , δ1 ], and χ(x) = 1 for |x| ≤ 2R, R > R0 . It is easy to see that h 1 Z i2 −1 tr eit (λ−Q j )h θ(t)ψ 2 (Q j )(1 − χ 2 ) dt j=1 2π h Z  
1 −1 2 = eitλh θ(t) tr ψ 2 (Q j ) j=1 e−it Q 2 / h (1 − χ 2 ) dt 2π h Z  2
1 −1 + eitλh θ(t) tr ψ 2 (Q 1 ) e−it Q j / h j=1 (1 − χ 2 ) dt = A + B . 2π h

424

BRUNEAU and PETKOV

This representation is justified by applying [2, Lem. 4.1], saying that

 2

2   

ψ (Q j ) 2 = O (h −n ),

ψ (Q 1 ) e−it Q j / h 2 = O (h −1−n ). j=1 tr j=1 tr We treat A below following closely the analysis of Bony in [2, Sec. 4.2]. Put A = A1 + A2 , where Z
1 −1 A1 = eitλh θ (t) tr (ψ(Q 1 ) − ψ(Q 2 ))e−it Q 2 / h ψ(Q 2 )(1 − χ 2 ) dt, 2π h Z
1 −1 A2 = eitλh θ (t) tr ψ(Q 1 )(ψ(Q 1 ) − ψ(Q 2 ))e−it Q 2 / h (1 − χ 2 ) dt. 2π h We deal only with the analysis of A1 since that of A2 is similar (see also [2, Sec. 4.2]). First, we find a pseudodifferential operator Q with symbol in S 0 (1) so that Z
1 −1 ˜ 2 ) dt, A1 = eitλh θ (t) tr e−it Q 2 / h ψ(Q 2 )Q(Q 1 − Q 2 )ψ(Q 2π h where ψ˜ ∈ C0∞ (R) is such that ψ˜ = 1 on supp ψ. We use the notation of [10] for h-pseudodifferential operators and set hxi = (1 + |x|2 )1/2 . Moreover, modulo a term in S N (1), the symbol of Q is supported in {(x, ξ ) : |x| > 2R}. Second, we obtain the existence of a pseudodifferential operator S with symbol
s(x, y, ξ ; h) ∈ S 0 hxi−n−1 hξ i−N , ∀N ∈ N, having compact support in ξ and (x − y) and support in {(x, ξ ) : |x| > 2R, (x, ξ ) ∈ l2−1 (I1 )} so that Z  1 −1 A1 = tr eitλh θ(t)e−it Q 2 / h S dt + O (h ∞ ). 2π h Applying [2, Th. 2], we obtain the existence of a Fourier integral operator Ut such that for |t| ≤ δ1 and δ1 sufficiently small we have kUt − e−it Q 2 / h Sktr = O (h ∞ ). R −1 Next, we write the kernel of the operator eitλh θ(t)Ut dt in the form Z Z 1 ei(tλ+8(t,x,ξ )−y·ξ )/ h θ(t)A(t, x, y, ξ ; h) dt dξ K (x, y; h) = (2π h)n and deduce that A1 =

1 (2π h)n+1

Z Z Z

ei(tλ+8(t,x,ξ )−x·ξ )/ h θ(t)A(t, x, x, ξ ; h) dt d x dξ + O (h ∞ ).

MEROMORPHIC CONTINUATION

425

Here 8(t, x, ξ ) is the solution of the eikonal equation ( ∂t 8 + l2 (x, ∂x 8) = 0, 8(0, x, ξ ) = x · ξ, l j (x, ξ ) being the principal symbol of Q j , j = 1, 2, and all derivatives
γ ∂tα ∂xβ ∂ξ 8(t, x, ξ ) − x · ξ are uniformly bounded for (t, x, ξ ) ∈ [−δ1 , δ1 ] × Rn × B(0, C1 ) and (α, β, γ ) 6= (0, 0, 0). Moreover, the symbol A(t, x, x, ξ ) has support in {(x, ξ ) : |x| > 2R, |ξ | ≤ C1 , (x, ξ ) ∈ l2−1 (I1 )} so that for all α and |t| ≤ δ1 we have |∂ α A(t, x, x, ξ )| ≤ Cα hxi−n−1 .

(A.1)

The last estimate enables us to calculate A1 by using an infinite partition of unity X 9(x − α) = 1, ∀x ∈ Rn , α∈Nn

9 ∈ C0∞ (K ), 9(x) ≥ 0, K being a neighborhood of the unit cube. Consequently, for every fixed h ∈ ]0, h 0 ] we have Z Z Z 1 lim ei(tλ+8(t,x,ξ )−x·ξ )/ h θ(t) A1 = (2π h)n+1 m→∞ X × 9(x − α)A(t, x, x, ξ ; h) dt d x dξ + O (h ∞ ) |α|≤m

= lim Im + O (h ∞ ), m→∞

and we reduce the problem to the analysis of the integrals Im over a compact set in (t, x, ξ ). Concerning the phase function, we observe that
tλ + 8(t, x, ξ ) − x · ξ = t λ − l2 (x, ξ ) + O (t) , where O (t) is uniformly bounded on the support of θ(t)A(t, x, x, ξ ) since the derivatives of (8(t, x, ξ ) − x · ξ ) are bounded on this set. Finally, to have a uniform bound for the remainder with respect to m → ∞, notice that |∂x,ξ l2 (x, ξ )| ≥ δ2 > 0

(A.2)

for |ξ | ≤ C1 , (x, ξ ) ∈ l2−1 (λ), λ ∈ I0 . The last condition follows easily from the form of the principal symbol X X l2 (x, ξ ) = |ξ |2 + bα,R (x)ξ α + bα,R (x)ξ α |α|=2

|α|≤1

426

BRUNEAU and PETKOV

of the operator Q 2 , constructed in [2], and the fact that |bα,R (x)| + |∂x bα,R (x)| ≤ 1 (R) with 1 (R) −→ 0 as R −→ +∞ (see [2, Sec. 2.3] for more details). Taking R  1 sufficiently large, we arrange (A.2) uniformly with respect to |ξ | ≤ C1 and (x, ξ ) ∈ l2−1 (λ). Now the critical points of the phase function (tλ + 8(t, x, ξ ) − x · ξ ) become t = 0, l2 (x, ξ ) = λ, and by the stationary phase method we obtain Im =

1 ψ(λ) (2π h)n Z X × 9(x − α)A1 (0, x, ξ, λ)(1 − χ 2 )(x)L λ (dω) + O (h 1−n ), l2 (x,ξ )=λ |α|≤m

where L λ (dω) is the Liouville measure on l2 (x, ξ ) = λ and the remainder O (h 1−n ) is uniform with respect to λ ∈ I0 and m ∈ N. Taking the limit limm→∞ Im , we obtain an asymptotics of A1 . For the analysis of B , we use the representation  −it Q / h 2 t j e = j=1 ih

1

Z

e−ist Q 1 / h (Q 1 − Q 2 )e−i(1−s)t Q 2 / h ds.

0

Following the argument in [2, Sec. 4.3], we find pseudodifferential operators

Q ∈ Oph S 0 (hxi−n−1 hξ i−N ) , Q˜ ∈ Oph S 0 (hξ i−N ) with symbols q(x, y, ξ ; h), q(x, ˜ y, ξ ; h) having compact support in ξ and (x − y) so that Z 1 Z  1 itλ/ h −ist Q 1 / h −i(1−s)t Q 2 / h ˜ B= tr e tθ(t) e Qe Q ds dt + O (h ∞ ). 2π h 2 0 Moreover, modulo a term in S N (1), the symbol of Q˜ is supported in {(x, ξ ) : |x| > 2R}. Applying an approximation of the unitary groups e−ist Q 1 / h , e−i(1−s)t Q 2 / h by Fourier integral operators, we are reduced to studying the integral J=

1 (2π h)2n+2

Z Z

1Z

eitλ/ h tθ(t)ei(81 (st,x,ξ )−z·ξ )/ h ei(82 ((1−s)t,z,η)−x·η)/ h

0

× B(t, s, X ) dt ds d X, where X = (x, z, ξ, η) and the phase functions 81 (t, x, ξ ) and 82 (s, z, η) are related to the eikonal equations with symbols l1 (x, ξ ) and l2 (z, η), respectively. The amplitude B(t, s, X ) has compact support with respect to (ξ, η), and its support with respect to x is included in the set {(x, ξ ) : |x| ≥ 2R}. Moreover, ∂ α B(t, s, X ) satisfy decreasing estimates with respect to (x, z) such as those in (A.1).

MEROMORPHIC CONTINUATION

427

In the same way as in [2], we check that the critical points of the phase in the integral J are related to the closed trajectories composed as the union of a curve  exp(τ Hl1 )(ρ) : 0 ≤ τ ≤ st of the Hamilton field Hl1 starting at some point ρ ∈ {(x, ξ ) ∈ Rn : |x| > 2R} and a curve  exp(τ Hl2 )(σ ) : 0 ≤ τ ≤ (1 − s)t , σ = exp(st Hl1 )(ρ), of the Hamilton field Hl2 . For 0 < t ≤ δ1 , δ1 sufficiently small, and R > 0 large enough, there are no such closed trajectories, and the critical points are obtained only for t = 0. We write the phase function in the form   t λ − sl1 (x, ξ ) − (1 − s)l2 (z, η) + O (t) + (x − z)(ξ − η), and the critical points become t = 0,

sl1 (x, ξ ) + (1 − s)l2 (x, ξ ) = λ,

x = z,

ξ = η.

For |x| ≥ 2R and 0 ≤ s ≤ 1, according to (2.6), we deduce m s (x, ξ ) = sl1 (x, ξ ) + (1 − s)l2 (x, ξ ) = |ξ |2 + η0 (R)|ξ |2 = l1 (x, ξ ) + η1 (R)|ξ |2 = l2 (x, ξ ) + η2 (R)|ξ |2 with ηi (R) −→ 0 as R → +∞, i = 0, 1, 2. Thus for λ ∈ I0 and R large enough the energy surface  6s (λ) = (x, ξ ) : m s (x, ξ ) = λ, |x| ≥ 2R is nondegenerate. Repeating the argument used for A1 , and applying the stationary phase method, we get an asymptotics Z 1Z 1 J= b(λ) B1 (s, x, ξ, λ)L s,λ (dω) ds + O (h 1−n ), (2π h)n 0 m s (x,ξ )=λ where L s,λ (dω) is the Liouville measure on 6s (λ). Notice that the first term with power h −1−n vanishes because we have the factor tθ(t), and the term involving h −n yields the contribution to the leading term in (5.10). Moreover, b(λ) has support in a small neighborhood of I1 , and taking R > 0 large, we may assume that b(λ) ∈ C00 (I0 ). This completes the proof of (5.10). The above argument shows that for λ ∈ / I0 the phase functions in Im and J have no critical points over the support of the integrand. Consequently, by an integration by parts, we obtain  2 tr (Fh−1 θ)(λ − Q j )ϕ2 (Q j )(1 − χ 2 ) j=1 = O (h ∞ ) uniformly with respect to λ ∈ / I0 .

428

BRUNEAU and PETKOV

Acknowledgments. The authors are grateful to J. Sj¨ostrand and M. Zworski for many helpful discussions. We also thank the referee for remarks. References [1] [2]

[3] [4] [5]

[6]

[7] [8] [9]

[10]

[11]

[12] [13]

[14] [15]

J.-F. BONY, R´esonances dans des domaines de taille h, Internat. Math. Res. Notices

2001, 817 – 847. MR 2002m:35237 391, 406, 413 , Minoration du nombre de r´esonances engendr´ees par une trajectoire ferm´ee, Comm. Partial Differential Equations 27 (2002), 1021 – 1078. CMP 1 916 556 391, 406, 410, 424, 426, 427 , Trajectoires ferm´ees et r´esonances, th`ese de doctorat, Universit´e Paris XI, 2001. 421 ¨ J.-F. BONY and J. SJOSTRAND , Traceformula for resonances in small domains, J. Funct. Anal. 184 (2001), 402 – 418. CMP 1 851 003 391, 392, 412, 415 V. BRUNEAU, Semi-classical behaviour of the scattering phase for trapping perturbations of the Laplacian, Comm. Partial Differential Equations 24 (1999), 1095 – 1125. MR 2000e:35172 392, 421, 423 V. BRUNEAU and V. PETKOV, Representation of the spectral shift function and spectral asymptotics for trapping perturbations, Comm. Partial Differential Equations 26 (2001), 2081 – 2119. MR 2002j:47080 392, 404, 410 N. BURQ, Semi-classical estimates for the resolvent in nontrapping geometries, Internat. Math. Res. Notices 2002, 221 – 241. MR 2002k:81069 419 N. BURQ and M. ZWORSKI, Resonance expansions in semi-classical propagation, Comm. Math. Phys. 223 (2001), 1 – 12. CMP 1 860 756 389 T. CHRISTIANSEN, Spectral asymptotics for compactly supported perturbations of the Laplacian on Rn , Comm. Partial Differential Equations 23 (1998), 933 – 948. MR 99j:35157 392, 404 ¨ M. DIMASSI and J. SJOSTRAND , Spectral Asymptotics in the Semi-Classical Limit, London Math. Soc. Lecture Note Ser. 268, Cambridge Univ. Press, Cambridge, 1999. MR 2001b:35237 397, 410, 424 M. DIMASSI and M. ZERZERI, A local trace formula for resonances of perturbed periodic Schr¨odinger operators, to appear in J. Funct. Anal., http://mpej.unige.ch/mp arc-bin/mpa?yn=01-484 393 S. FUJIIE and T. RAMOND, Breit-Wigner formula at barrier tops, preprint, 2002, http://mpej.unige.ch/mp arc-bin/mpa?yn=02-210 412 ´ C. GERARD, A. MARTINEZ, and D. ROBERT, Breit-Wigner formulas for the scattering phase and the total scattering cross-section in the semi-classical limit, Comm. Math. Phys. 121 (1989), 323 – 336. MR 90b:35185 412 L. GUILLOPE´ and M. ZWORSKI, Scattering asymptotics for Riemann surfaces, Ann. of Math. (2) 145 (1997), 597 – 660. MR 98g:58181 389, 390 T. E. GUREEV and YU. G. SAFAROV, Exact spectral asymptotics for the Laplace operator on a manifold with periodic geodesics (in Russian), Trudy Mat. Inst. Steklov. 179 (1988), 36 – 53; English translation in Proc. Steklov Inst. Math. 1989, no. 2, 35 – 53. MR 89i:35106 420

MEROMORPHIC CONTINUATION

429

[16]

A. MARTINEZ, Resonance free domains for non globally analytic potentials, Ann.

[17]

R. B. MELROSE, Weyl asymptotics for the phase in obstacle scattering, Comm. Partial

[18]

S. NAKAMURA, Spectral shift function for trapping energies in the semiclassical limit,

[19]

L. NEDELEC, Resonances for matrix Schr¨odinger operators, Duke Math. J. 106 (2001),

[20]

V. PETKOV and G. POPOV, Semi-classical trace formula and clustering of eigenvalues

Henri Poincar´e 3 (2002), 739 – 756. CMP 1 933 368 419 Differential Equations 13 (1988), 1431 – 1439. MR 90a:35183 389, 392 Comm. Math. Phys. 208 (1999), 173 – 193. MR 2000j:81066 392 209 – 236. MR 2002a:35046 393

[21]

[22]

[23]

[24] [25] [26]

[27]

[28]

[29]

[30]

[31]

for Schr¨odinger operators, Ann. Inst. H. Poincar´e Phys. Th´eor. 68 (1998), 17 – 83. MR 99c:47073 420 V. PETKOV and D. ROBERT, Asymptotique semi-classique du spectre d’hamiltoniens quantiques et trajectoires classiques p´eriodiques, Comm. Partial Differential Equations 10 (1985), 365 – 390. MR 86m:35130 V. PETKOV and M. ZWORSKI, Breit-Wigner approximation and the distribution of resonances, Comm. Math. Phys. 204 (1999), 329 – 351, MR 2000j:81279; Erratum, Comm. Math. Phys. 214 (2000), 733 – 735. MR 2002a:81324 389, 390, 412, 420 , Semi-classical estimates on the scattering determinant, Ann. Henri Poincar´e 2 (2001), 675 – 711. MR 2002h:35222 389, 390, 391, 392, 393, 412, 413, 414, 415, 417, 418 D. ROBERT, Autour de l’approximation semi-classique, Progr. Math. 68, Birkh¨auser, Boston, 1987. MR 89g:81016 397, 410 , Relative time-delay for perturbations of elliptic operators and semiclassical asymptotics, J. Funct. Anal. 126 (1994), 36 – 82. MR 95j:35162 392, 404 YU. G. SAFAROV, Asymptotics of the spectrum of a pseudodifferential operator with periodic bicharacteristics (in Russian), Zap. Nauchn. Sem. Leningrad. Otdel Math. Inst. Steklov. (LOMI) 152 (1986), 94 – 104; English translation in J. Soviet Math. 40 (1988), 645 – 652. MR 88d:58130 420, 422 ¨ J. SJOSTRAND , A trace formula for resonances and application to semi-classical ´ ´ Schr¨odinger operators, S´emin. Equ. D´eriv. Partielles 1996 – 1997, Ecole Polytech., Palaiseau, 1997, exp. no. 2. MR 98k:35143 406 , “A trace formula and review of some estimates for resonances” in Microlocal Analysis and Spectral Theory (Lucca, Italy, 1996), NATO Adv. Sci. Inst. Ser. C Math. Phys. Sci. 490, Kluwer, Dordrecht, 1997, 377 – 437. MR 99e:47064 389, 390, 391, 392, 393, 395, 396, 397, 399, 402, 417 , Resonances for bottles and trace formulae, Math. Nachr. 221 (2001), 95 – 149. MR 2001k:58063 389, 391, 392, 393, 395, 396, 398, 399, 401, 402, 403, 412, 417 ¨ J. SJOSTRAND and M. ZWORSKI, Complex scaling and the distribution of scattering poles, J. Amer. Math. Soc. 4 (1991), 729 – 769. MR 92g:35166 389, 391, 393, 399 P. STEFANOV, Sharp upper bounds on the number of resonances near the real axis for trapping systems, to appear in Amer. J. Math. 393, 416, 423

430

BRUNEAU and PETKOV

[32]

S.-H. TANG and M. ZWORSKI, Resonance expansions of scattered waves, Comm. Pure

[33]

D. R. YAFAEV, Mathematical Scattering Theory, Transl. Math. Monogr. 105, Amer.

[34]

´ M. ZWORSKI, Poisson formulae for resonances, S´emin. Equ. D´eriv. Partielles

Appl. Math. 53 (2000), 1305 – 1334. MR 2001f:35306 389, 419 Math. Soc., Providence, 1992. MR 94f:47012 389, 421

[35]

´ 1996 – 1997, Ecole Polytech., Palaiseau, 1997, exp. no. 13. MR 98j:35036 389, 390 , Poisson formula for resonances in even dimensions, Asian J. Math. 2 (1998), 609 – 617. MR 2001g:47091 389, 390

Bruneau D´epartement de Math´ematiques Appliqu´ees, Universit´e Bordeaux I, 351, cours de la Lib´eration, 33405 Talence CEDEX, France; [email protected] Petkov D´epartement de Math´ematiques Appliqu´ees, Universit´e Bordeaux I, 351, cours de la Lib´eration, 33405 Talence CEDEX, France; [email protected]

EQUIVARIANT DEFORMATION OF MUMFORD CURVES AND OF ORDINARY CURVES IN POSITIVE CHARACTERISTIC GUNTHER CORNELISSEN and FUMIHARU KATO

Abstract We compute the dimension of the tangent space to, and the Krull dimension of, the prorepresentable hull of two deformation functors. The first one is the “algebraic” deformation functor of an ordinary curve X over a field of positive characteristic with prescribed action of a finite group G, and the data are computed in terms of the ramification behaviour of X → G\X . The second one is the “analytic” deformation functor of a fixed embedding of a finitely generated discrete group N in PGL(2, K ) over a nonarchimedean-valued field K , and the data are computed in terms of the Bass-Serre representation of N via a graph of groups. Finally, if 0 is a free subgroup of N such that N is contained in the normalizer of 0 in PGL(2, K ), then the Mumford curve associated to 0 becomes equipped with an action of N / 0, and we show that the algebraic functor deforming the latter action coincides with the analytic functor deforming the embedding of N . Introduction Equivariant deformation theory is the correct framework for formulating and answering questions such as the following: Given a curve X of genus g over a field k and a finite group of automorphisms ρ : G ,→ Aut(X ) of X , in how many ways can X be deformed into another curve of the same genus on which the same group of automorphisms still acts? The precise meaning of this question (at least infinitesimally) is related to the deformation functor D X,ρ of the pair (X, ρ), which associates to any element A of the category Ck of local Artinian k-algebras with residue field k the set of isomorphism classes of liftings (X ∼ , ρ ∼ , φ ∼ ), where X ∼ is a smooth scheme of finite type over A, φ ∼ is an isomorphism of X ∼ ⊗ k with X , and ρ ∼ : G → Aut A (X ) lifts ρ via φ ∼ . In general, D X,ρ has a prorepresentable hull H X,ρ in the sense of M. Schlessinger [18]. This means that there is a smooth map of functors Hom(H X,ρ , −) → D X,ρ which induces an isomorphism on the level of DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 3, Received 29 March 2001. Revision received 5 November 2001. 2000 Mathematics Subject Classification. Primary 14G22, 14D15, 14H37.

431

432

CORNELISSEN and KATO

tangent spaces, where H X,ρ is a Noetherian complete local k-algebra with residue field k. The above question can then be reformulated as the computation of the Krull dimension of H X,ρ . Two remarks are in order. First, if g ≥ 2, then the functor is even prorepresentable by H X,ρ , so Spf H X,ρ is a formal scheme that is, so to speak, the universal basis of a family of curves that have the same automorphism group as X . Second, H X,ρ is even algebraizable (cf. A. Grothendieck [11, Section 3]), and the underlying algebraic scheme over k might be considered as the genuine “universal basis” scheme. If the characteristic of the ground field k is zero, the dimension of H X,ρ is easy to compute. All obstructions and group cohomology (cf. Section 3.2) disappear, and we find that dim H X,ρ = 3gY − 3 + n, (1) where Y := G\X is the quotient of X , gY is its genus, and n is the number of branch points on Y . (Note that 3gY − 3 is the degree of freedom of varying the moduli of Y and that one extra degree of freedom comes in for every branch point.) This result can be found in any classical text on Riemann surfaces (see, e.g., [7, Section V.2.2]); the moral is that ramification data of X → Y provide all the necessary information for computing H X,ρ . In this work we are interested in the corresponding question in positive characteristic. Let us first present the motivating example for our studies: moduli schemes for rank 2 Drinfeld modules with principal level structure (see, e.g., E.-U. Gekeler and M. Reversat [8]). Example Let q = p t , F = Fq (T ), and A = Fq [T ]; let F∞ = Fq ((T −1 )) be the completion of F, and let C be a completion of the algebraic closure of F∞ . On Drinfeld’s “up1 − P1 (which is a rigid analytic space over C), the group per half-plane”  := PC F∞ GL(2, A) acts by fractional transformations. Let Z ∼ = Fq∗ be its center. For n ∈ A, the quotients of  by congruence subgroups 0(n) = {γ ∈ GL(2, A) : γ = 1 mod n} are open analytic curves that can be compactified to projective curves X (n) by adding finitely many cusps. These curves are analogues in the function field setting of classical modular curves X (n) for n ∈ Z. Clearly, elements from G(n) := 0(1)/ 0(n)Z induce automorphisms of X (n). It is even known that G(n) is the full automorphism group of X (n) if p 6= 2, q 6= 3 (cf. [5, Proposition 4]). It follows from (1) that a classical modular curve does not admit equivariant deformations since X (n) → X (1) = P1 is ramified above three points. (The most famous such curve is probably X (7), which is isomorphic to Klein’s quartic of genus 3 with PSL(2, 7) as the automorphism group.) What is the analogous result for the Drinfeld modular curves X (n)? Note that X (n) → X (1) = P1 is ramified above two points with ramification groups

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

433

Z/(q + 1)Z and (Z/ pZ)td o Z/(q − 1)Z, where d = deg(n) (cf. [5, Proposition 4]), which already shows that something goes wrong when naively applying (1). The correct framework for carrying out such computations is that of equivariant cohomology (see Grothendieck [10]). It predicts that in positive characteristic the group cohomology of the ramification groups with values in the tangent sheaf at branch points contribute to the deformation space. J. Bertin and A. M´ezard have considered the case of a cyclic group of prime order G = Z/ pZ in [1], mainly concentrating on mixed characteristic lifting. In this paper, we work in the equicharacteristic case and we do not want to impose direct restrictions on the group G, but rather on the curve X , which we require to be ordinary. This means that the p-rank of its Jacobian satisfies dimF p Jac(X )[ p] = g. The property of being ordinary is open and dense in the moduli space of curves of genus g, so our calculations do apply to a large portion of that moduli space. (There is some cheating here since curves without automorphisms are also dense; but notice that the analytic constructions in part B of the paper at least show the existence of lots of ordinary curves with automorphisms.) The main advantage of working with ordinary curves is that their ramification groups are of a very specific form, so the Galois cohomological computation becomes feasible (cf. Proposition 1.4): If Pi is a point on Y branched in X → Y , then G Pi ∼ = (Z/ pZ)ti o Z/n i Z

(2)

for some integers (ti , n i ) satisfying n i | p ti − 1. If ti > 0, we say that Pi is wildly branched. Our main algebraic result is the following (which we state here in a form that excludes a few anomalous cases); one can think of it giving the “positive characteristic error term” to the Riemann surface computation in (1). (cf. Theorem 5.1) Assume that p 6= 2, 3, that X is an ordinary curve of genus g ≥ 2 over a field of characteristic p > 0, and that G is a finite group acting via ρ : G → Aut(X ) on X , such that X → Y := G\X is branched above n points, of which the first s are wildly branched. Then the equicharacteristic deformation functor D X,ρ is prorepresentable by a ring H X,ρ whose Krull dimension is given by MAIN ALGEBRAIC THEOREM

dim H X,ρ = 3gY − 3 + n +

s X i=1 0

ti , s(n i )

where s(n i ) := [F p (ζn i ) : F p ] = min{s 0 > 0 : n i | p s − 1}, gY is the genus of Y , and (ti , n i ) are the data corresponding to the wild ramification points via (2).

434

CORNELISSEN and KATO

The theorem is proved by first computing the first-order local deformation functors, then studying their liftings to all of Ck , and putting the results together via the localization theorem [1, Th´eor`eme 3.3.4]. In the end, we can even describe the ring H X,ρ explicitly (cf. Proposition 1.12 and Section 4.4). It turns out to be a formal polydisc to which, for each ramification point with n i ≤ 2, a (( p − 1)/2)-nilpotent zero-dimensional scheme is attached. (In characteristic p = 2 even more weird things can happen.) These nilpotent schemes are related to the lifting of a specific first-order deformation via what we call formal truncated Chebyshev polynomials. Example (continued) For the Drinfeld modular curve X (n), we find a (d − 1)-dimensional reduced deformation space. The second part of this paper is concerned with analytic equivariant deformation theory. Let (K , | · |) be a nonarchimedean-valued field of positive characteristic p > 0 with residue field k. Recall that a Mumford curve over K is a curve X whose stable reduction is isomorphic to a union of rational curves intersecting in k-rational points. D. Mumford has shown that this is equivalent to its analytification X an being isomoran phic to an analytic space of the form 0\(P1, K − L0 ), where 0 is a discontinuous group in PGL(2, K ) with L0 as the set of limit points. It is known that Mumford curves are ordinary, so the above algebraic theory applies. But what interests us most in this second part is to find out where these deformations “live” in the realm of discrete groups. The setup is as follows: Let X be a Mumford curve, and let N be a group contained in the normalizer N (0) of the corresponding so-called Schottky group 0 such that 0 ⊆ N . Then there is an injection ρ : G := N / 0 ,→ Aut(X ) (an isomorphism if N = N (0)). By rigidity, two Mumford curves are isomorphic if and only if their Schottky groups are conjugate in PGL(2, K ). Hence it is natural to consider the analytic deformation functor D N ,φ : C K → Set that associates to A ∈ C K the set of homomorphisms N → PGL(2, A) that lift the given morphism φ in the obvious sense. This functor comes with a natural action of conjugation by the group functor PGL(2)∧ : C K → Groups given by   PGL(2)∧ (A) := ker PGL(2, A) → PGL(2, K ) , ∼ ∧ and we denote by D ∼ N ,φ the quotient functor D N ,φ = PGL(2) \D N ,φ . We now set out to compute the hulls corresponding to these functors. For this we use the fact that N can be described by the theorem of H. Bass and J.-P. Serre, which states that there is a graph of groups (TN , N• ) such that N is a semidirect product of a free group (of rank the cyclomatic number of TN ) and the tree (amalgamation) product associated to a lifting of stabilizer groups N∗ of vertices

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

435

and edges ∗ from TN to the universal covering of TN as as graph. In a technical proposition, we show that this decomposition of the group N induces a decomposition of functors D N ,φ ∼ = lim D N• ,φ| N• × D Fc ,φ◦s , ←− TN

where s is a section of the semidirect product. (Note that this is a direct product of functors, that the inverse limit is over TN , not over its universal covering, and that we are using D N ,φ , not D ∼ N ,φ .) This reduces everything to the computation of D N ,φ for free N (which is easy) or finite N acting on P1 . The latter can be done by using the classification of finite subgroups of P1 and the algebraic results from the first part in the particular case of P1 . Modulo a few anomalous cases, the result is the following. MAIN ANALYTIC THEOREM (cf. Theorem 8.4) If X is a Mumford curve of genus g ≥ 2 over a nonarchimedean field K of characteristic p > 3 with Schottky group 0, then for a given discrete group N contained in the normalizer of 0 in PGL(2, K ) with corresponding graph of groups (TN , N• ), the equicharacteristic analytic deformation functor D ∼ N ,φ is prorepresentable by a ring H N∼,φ whose dimension satisfies

dim H N∼,φ = 3c(TN ) − 3 +

X v=vertex of T N

d(v) −

X

d(e),

e=edge of T N

where c(∗) denotes the cyclomatic number of a graph ∗ and   2 if N∗ = Z/nZ,    3 if N∗ = A4 , S4 , A5 , Dn , PGL(2, p t ), PSL(2, p t ), d(∗) =  t if N∗ = (Z/ pZ)t ,     t/s(n) + 2 if N∗ = (Z/ pZ)t o Z/nZ, where n is coprime to p. Example (continued) The Drinfeld modular curve X (n) is known to be a Mumford curve, and the normalizer of its Schottky group is isomorphic to an amalgam (cf. [5, Proposition 4]) N (n) = PGL(2, q) ∗(Z/ pZ)t oZ/(q−1)Z (Z/ pZ)td o Z/(q − 1)Z.

(3)

The above formula again gives a (d − 1)-dimensional deformation space. We can see these deformations explicitly as follows. By conjugation with PGL(2, C), we can assume that the embedding of PGL(2, p t ) in (3) into PGL(2, C) is induced by the

436

CORNELISSEN and KATO

standard Fq ⊆ C. Then a little matrix computation shows that the freedom of choice left is that of a d-dimensional Fq -vector space V of dimension d in C which contains the standard Fq in order to embed the order- p elements from the second group
involved in the amalgam as 10 x1 for x ∈ V . In the final section of the paper, we compare algebraic and analytic deformation functors. What comes out is an isomorphism of functors ∼ D∼ N ,φ = D X,ρ . To prove this, we remark that the whole construction of Mumford can be carried out in the category C K (by fixing a lifting of N to PGL(2, A)) to produce a scheme X 0 over Spec A whose central fiber is isomorphic to X , and such that X 0 carries an action ρ of N / 0 which reduces to the given action on X . Remark. We did not take the more global road of “equivariant Teichm¨uller space” to analytic deformation, as F. Herrlich does in [12] and [13]; one can consider the space
M (N , 0) = PGL(2, K )\ Hom∗ N , PGL(2, K ) / Aut0 (N ), where Hom∗ means the space of injective morphisms with discrete image, the action of PGL(2, K ) from the left is by conjugation, and the action of the group of automorphisms of N which fix 0 is on the right. The relations in N impose a natural structure on M (N , 0) as an analytic space, but in positive characteristic this structure might be nonreduced due to the presence of parabolic elements; for example, if N contains
Z/ pZ, p > 3, then the condition that a matrix γ = ac db be of order p leads to
( p−1)/2 tr2 (γ ) − 4 det(γ ) · b = 0. Therefore in [13] global considerations are restricted to the case where N does not contain parabolic elements (see [13, p. 148]), whereas our local calculations are independent of such restrictions. Let us note that if K has characteristic zero, then in [13] a formula is given for the dimension of M (N , 0) (which turns out to be an equidimensional space) compatible with our main analytical theorem. Convention. Throughout this paper, if R is a local ring, we let dim R denote its Krull dimension, and if V is a k-vector space, we let dimk V denote its dimension as a k-vector space.

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

437

Part A: Algebraic theory 1. Deformation of Galois covers 1.1. The global deformation functor We start by recalling the definition of an equivariant deformation of a curve with a given group of automorphisms (see [1] for an excellent survey). Let k be a field, and let X be a smooth projective curve over k. We fix a finite subgroup G in the automorphism group Autk (X ) of X , and we denote by ρ the inclusion ρ : G ,→ Autk (X ) : σ → ρσ . Let Y = G\X be the quotient curve, and denote by π the quotient map X → Y . Let Ck be the category of Artinian local k-algebras with the residue field k. A lifting of (X, ρ) to A is a triple (X ∼ , ρ ∼ , φ ∼ ) consisting of a scheme X ∼ which is smooth of finite type over A, an injective group homomorphism ρ ∼ : G ,→ Aut A (X ∼ ) : σ → ρσ∼ , and an isomorphism φ ∼ : X ∼ ⊗ Spec A Spec k → X of schemes over k such that ρ ∼ = ρ. Here, ρ ∼ denotes the composite of ρ ∼ and the restriction onto the central fiber, identified with X by φ ∼ . Two liftings (X ∼ , ρ ∼ , φ ∼ ) and (X ≈ , ρ ≈ , φ ≈ ) are said to be isomorphic if there exists an isomorphism ψ : X ≈ → X ∼ of schemes over A such that φ ∼ ◦ (ψ ⊗ A k) = φ ≈ and, for any σ ∈ G, ψ ◦ ρσ≈ = ρσ∼ ◦ ψ. We arrive at the deformation functor D X,ρ : Ck → Set that assigns to any A ∈ Ck the set of isomorphism classes of liftings of (X, ρ). 1.2. The functor π∗G on tangents In order to compute the tangent space to this deformation functor, we need to recall the equivariant cohomology theory of Grothendieck. The morphism π induces a tangent map T X → π ∗ TY , which is a monomorphism since π is generically e´ tale. (Dually,  X/Y is a torsion sheaf.) Note that both T X and π ∗ TY are G-O X -modules (cf. [10, Section 5.1]) and that the tangent map T X → π ∗ TY is a morphism of G-O X -modules; the G-structure of π ∗ TY = N O X OY π −1 TY is the tensor product of the usual G-structure on O X and the trivial G-structure on π −1 TY (cf. [10, Section 5.1]). We apply the functor π∗G to the tangent

438

CORNELISSEN and KATO

map; recall that, by definition, for a G-sheaf F on X and an open U of Y , π∗G (F )(U ) is the set of sections of F on π −1 (U ) invariant under G. Since π∗G π ∗ TY ∼ = TY (cf. [10, (5.1.1)]), we get 0 → π∗G T X −→ TY . (1.2.1) The fact that this map is a monomorphism is due to the left exactness of π∗G . Note that π∗G T X is an invertible sheaf on Y ; indeed, it can be described by formula (1.6.1). 1.3. Ordinary curves From now on, we assume that the field k is of characteristic p > 0 and that the curve X is ordinary. This means that dimF p Jac(X )[ p] = g. The property of being ordinary is open and dense on the moduli space of curves of genus g (see [17, Section 4], [14]) since it is essentially a maximal rank condition on the Hasse-Witt matrix of the curve (hence open), and ordinary curves exist for all g. 1.4 Let X be an ordinary curve, let G be a finite group of automorphisms of X , let π : X → G\X , and let P ∈ Y be a branch point of π. Then the ramification filtration for P stops at G 2 = {1}. The ramification group at P is of the form (Z/ pZ)t o Z/nZ for n| p t − 1. Actually, letting s := [F p (ζ ) : F p ], where ζ is a primitive nth root of unity, one can consider (Z/ pZ)t as a vector space of dimension t/s over Fq , where q = p s , and the action of Z/nZ is exactly by multiplication with ζ . Locally at P, the cover can be decomposed as a tower



k (xt/s ) − · · · − k (x2 ) − k (x1 ) − k (x0 ) , PROPOSITION

where the first step is a Kummer extension of degree n (x0 = x1n ) and all others are −q −1 Artin-Schreier extensions of degree q (xi − xi−1 = ci xi−1 ). Proof Let P be a ramified point in that cover, and let G 0 ⊇ G 1 ⊇ G 2 ⊇ · · · be the standard filtration on the ramification group G 0 at P. In [17, Theorem 2(i)], S. Nakajima has shown that for an ordinary curve, G 2 = {1}. It is also known that G i /G i+1 for i ≥ 1 are elementary abelian p-groups and that G 0 is the semidirect product of a group of order prime to p and G 1 with the action given above (see [19, Section IV.2, Corollaries 1 – 4]). The results follows from this and standard field theory.

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

439

Remark 1.4.1 The following formula for s holds (see [15, Section 2.47(ii)]): 0

s = min{s 0 ∈ Z>0 : n | p s − 1}. Remark 1.4.2 The calculations that follow actually depend only upon the special form of the ramification filtration given in Proposition 1.4, so they also apply to (not necessarily ordinary) curves for which the action of G is as described here. Notation 1.5 We break up the branch points P of π, which we enumerate as {P1 , . . . , Pn }, into two sets T and W ; let Ztp o Zn be the ramification group at P. Then P ∈ T ⇐⇒ t = 0 or ( p = 2 and t = 1), and P ∈ W otherwise. Note that for p 6= 2, points in T are called tame and points in W are called wild; but for p = 2 the distinction is more subtle. Observe that p = 2 and t = 1 implies n = 1. We define the divisor 1 on Y by P P 1= P +2 Q of degree δ := deg(1) = |T | + 2|W |. P∈T

Q∈W

PROPOSITION 1.6 We have π∗G T X ∼ = TY (−1).

Proof As in [1, proof of Proposition 5.3.2], we have 
 π∗G T X ∼ = TY ⊗ OY ∩ π∗ O X (−r) ,

(1.6.1)

where r is the (global) different of k(X )/k(Y ). The divisor r is supported at branch P∞ points P. The exponent of the local different at P is i=0 (|G i | − 1), where G i are the higher ramification groups at P. Then r P = (np t − 1 + p t − 1) · P follows from Proposition 1.4. Hence, upon intersecting with OY in (1.6.1), we get (recall that the cover is totally ramified)  l 
 pt − 2 m  OY ∩ π∗ O X (−r) P = O − 1 + P np t ( O (−P) if t = 0 or ( p = 2, t = 1), = O (−2P) otherwise, where dxe = min{n ∈ Z>0 : n ≥ x}. If we now collect the local terms, the result comes out.

440

CORNELISSEN and KATO

Remark 1.6.2 In [1, pp. 235 – 236], d·e and b·c have to be interchanged everywhere. 1.7. Cohomology of π∗G We now recall the two spectral sequences from [10, Section 5.2]. Let F be a coherent G-O X -module, and write Hi (X ; G, F ) = Ri 0 G X F, H i (G, F ) = Ri π∗G F , G G G where 0 G X = 0Y ◦π∗ , that is, where 0 X F = (0 X F ) . There are two cohomological spectral sequences
I E p,q = H p Y, H q (G, F ) =⇒ H p+q (X ; G, F ), 2
I I E p,q = H p G, Hq (X, F ) =⇒ H p+q (X ; G, F ). 2

The first one gives rise to the edge sequence
0 −→ H1 (Y, π∗G F ) −→ H1 (X ; G, F ) −→ H0 Y, H 1 (G, F ) , −→ 0 (1.7.1) as we are on a curve Y . 1.8. Tangent space to the global deformation functor We can use this equivariant cohomology to compute the tangent space to our deformation functor. Recall that the ring of dual numbers is defined as k[] = k[E]/(E 2 ); clearly, it belongs to Ck . The tangent space to the deformation functor D X,ρ is by definition its value on the ring of dual numbers with its natural k-linear structure (cf. [18]). PROPOSITION 1.9 (see [1, (3.2.1), (3.3.1)]) We have D X,ρ (k[]) ∼ = H1 (X ; G, T X ).

1.10. Localization We now describe localization for our deformation functor. Let Q j ∈ X be a point lying over P j for 1 ≤ j ≤ n, and let G j be the stabilizer of Q j , which acts on the local ring O X,Q j by k-algebra automorphisms; we denote it by ρ j : G j → Autk (O X,Q j ). Changing the choice of Q j does not affect, up to a suitable equivalence, the action ρ j . Every lifting (X ∼ , ρ ∼ , φ ∼ ) induces a lifting of the local representation ρ j for any 1 ≤ j ≤ n in the sense defined in Section 1.11.

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

441

1.11. The local deformation functor Let R be a k-algebra, and let G be a group acting via ρ : G → Autk (R) on R by k-algebra automorphisms. Let A ∈ Ck , and let R A = R ⊗k A. Let · denote the reduction map R A → R modulo the maximal ideal m A of A. We define a lifting of ρ to A as a group homomorphism ρ ∼ : G → Aut A (R A ) such that ρ ∼ = ρ. Two liftings ρ ∼ and ρ ≈ are said to be isomorphic (over ρ) if there exists ψ ∈ Aut A (R A ) with ψ = Id R such that for any σ ∈ G, ψ ◦ ρσ≈ = ρσ∼ ◦ ψ. We arrive at a local deformation functor Dρ : Ck −→ Set such that Dρ (A) is the set of all isomorphism classes of liftings of ρ to A. It is but a formal check using Schlessinger’s criterion to see that Dρ has a prorepresentable hull (which we denote by Hρ ) if G is a finite group (cf. [1, Th´eor`eme 2.2]). In our situation we get a transformation of functors D X,ρ −→ Dρ1 × · · · × Dρn .

(1.11.1)

It turns out that this morphism is formally smooth (see [1, Th´eor`eme 3.3.4]). Hence we get the following localization result. PROPOSITION 1.12 ([1, Lemme 3.3.1 and Corollaire 3.3.5]) The functor D X,ρ has a prorepresentable hull H X,ρ ; in fact,

ˆ · · · ⊗H ˆ ρn [[u 1 , . . . , u N ]], H X,ρ ∼ = Hρ1 ⊗ where N = dimk H1 (Y, π∗G T X ). Remark 1.12.1 As soon as g ≥ 2, the curve X does not have infinitesimal automorphisms, so the functor D X,ρ is in fact prorepresentable by H X,ρ (cf. [1, Th´eor`eme 2.1]). 2. Lifting of group actions and group cohomology The aim of Sections 2 – 4 is to compute the tangent space and the prorepresentable hull of the local deformation functors for the representations of the branch groups in quotients of ordinary curves as automorphisms of the local stalks of the tangent sheaf. 2.1. Action on derivations Let k be a field, and let R be a k-algebra. We denote by T R the R-module of kderivations, that is, the set of all maps δ : R → R such that δ(x y) = δ(x)y + xδ(y) for x, y ∈ R and such that δ(a) = 0 for a ∈ k. Let ϕ ∈ Autk (R) be an automorphism

442

CORNELISSEN and KATO

of R over k. Then it induces a k-module automorphism of T R , denoted by Adϕ , by Adϕ (δ) = ϕ ◦ δ ◦ ϕ −1 . Note that Adϕ is not an R-module automorphism but rather is equivariant; for x ∈ R and δ ∈ T R , we have Adϕ (xδ) = ϕ(x) Adϕ (δ). Let G be a finite group, and suppose it acts on R by k-algebra automorphisms; that is, suppose a group homomorphism ρ : G → Autk (R) : σ 7→ ρσ is given. The induced action of G on T R by k-module automorphisms is denoted by Adρ : σ 7→ Adρ,σ . 2.2. Tangent space to the local deformation functor Let k[] be the ring of dual numbers. Then R[] = R ⊗k k[] is the k[]-algebra of elements x + y with x, y ∈ R. Let · : R[] → R denote the reduction map modulo . In the following proposition, we make an explicit identification between the tangent space Dρ (k[]) to the deformation functor Dρ and the first group cohomology with values in the derivations. PROPOSITION 2.3 There exists a bijection (depending on the deformation parameter ) ∼ d : Dρ (k[]) −→ H1 (G, T R ) described as follows: The 1-cocycle dρ ∼ associated to a lifting ρ ∼ is given by the formula

dρ ∼ σ =

ρσ∼ ◦ ρσ−1 − Id 



=

 d ∼ (ρσ ◦ ρσ−1 )|=0 d

for any σ ∈ G. Proof For σ ∈ G we set ρσ∼ (x) = ρσ (x) + ρσ0 (x) (x ∈ R). Then for x + y ∈ R[], we have   ρσ∼ (x + y) = ρσ∼ (x) + ρσ∼ (y) = ρσ (x) + ρσ0 (x) + ρσ (y) ; that is, ρσ0 determines the lifting ρ ∼ . The 1-cocycle dρ ∼ is given by dρ ∼ σ = ρσ0 ◦ ρσ−1 . The following two formulas are straightforward: (i) ρσ0 (x y) = ρσ (x)ρσ0 (y) + ρσ0 (x)ρσ (y) for x, y ∈ R, (ii) ρσ0 τ = ρσ0 ◦ ρτ + ρσ ◦ ρτ0 for σ, τ ∈ G. From (i) it follows that dρ ∼ σ ∈ T R , and from (ii) it follows that dρ ∼ is a cocycle; that is, dρ ∼ σ τ = dρ ∼ σ + Adρ,σ (dρ ∼ τ ).

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

443

Suppose that two liftings ρ ∼ and ρ ≈ are isomorphic by ψ ∈ Autk[] (R[]). By the condition ψ = Id R , we can write ψ(x) = x + δ(x) (x ∈ R), where δ ∈ T R . The equality ψ ◦ ρσ≈ = ρσ∼ ◦ ψ implies ρσ00 − ρσ0 = ρσ ◦ δ − δ ◦ ρσ ,

(2.3.1)

where ρσ00 is the -part in ρ ≈ (as ρσ0 was the -part of ρ ∼ ). Hence dρ ≈ σ − dρ ∼ σ = Adρ,σ (δ) − δ,

(2.3.2)

which implies that (dρ ≈ ) − (dρ ∼ ) is a coboundary. Conversely, if we have (2.3.2) for some δ ∈ T R , then one can define ψ ∈ Autk[] (R[]) by the obvious formula ψ(x) = x + δ(x) (or, equivalently, ψ(x + y) = x + [y + δ(x)]), which gives an isomorphism between the liftings ρ ∼ and ρ ≈ . Therefore the map d is well defined and is injective. We now show surjectivity. A given cocycle d : G → T R induces an automorphism ρσ∼ for any σ ∈ G by the formula ρσ∼ (x) = ρσ (x) + (dσ ) ◦ ρσ (x) for x ∈ R. One can easily check that σ 7→ ρσ∼ gives a lifting of ρ whose associated 1-cocycle is exactly d. 3. Computation of group cohomology 3.1 In this section, k denotes a field of characteristic p > 0, and O denotes a discrete valuation ring over k with the residue field k. We fix a regular parameter x for O . The O -module TO (defined in (2.1)) is free of rank 1 since every δ ∈ TO is determined by δ(x). Let ddx be the unique k-derivation such that ddx (x) = 1. Then TO = O ddx . Let a group G act on O by k-algebra automorphisms. In this section we write the action of ρ exponentially ( f → f σ for f ∈ O and σ ∈ G), omitting ρ from the notation—we do the same for the induced action Ad on the tangent space TO . The G-equivariancy condition now becomes ( f δ)σ = f σ δ σ for f ∈ O . In this section we compute the group cohomology H1 (G, TO ) in the following three situations, which are exactly the ones that arise for the local action of the ramification group of a branch point on an ordinary curve. Case 1. We have the tame case, so G = hτ i ∼ = Z/nZ with (n, p) = 1 acting on O by x τ = ζ x, where ζ is a primitive nth root of unity. Qt Case 2. We have the p-group case, so G = i=1 hσi i ∼ = (Z/ pZ)t acting on O by σ x i = x/(1 − u i x), where u 1 , . . . , u t ∈ k are linearly independent over F p . Let V be the t-dimensional F p -vector subspace in k spanned by u 1 , . . . , u t . The group G and

444

CORNELISSEN and KATO

its action are isomorphic to the vector group V , acting on O by x u = x/(1 − ux) for u ∈ V ; by this, we can (and do) pretend that G = V . Qt Case 3. We have the mixed case, so G = N o H , where N = i=1 hσi i ∼ = (Z/ pZ)t t ∼ and H = hτ i = Z/nZ with n > 1 and n| p − 1. Let ζ be a primitive nth root of unity in k, and let s = [F p (ζ ) : F p ] and q = p s as in Proposition 1.4. Similarly to the previous case, we consider N as as vector group V of dimension t/s over Fq acting on O by x u = x/(1 − ux) for u ∈ V , and the action of H is scalar multiplication by ζ ∈ Fq∗ on x. 3.2. Case 1 Since the G-module TO is killed by p but p is prime to the order of G, all higher group cohomology vanishes: Hn (G, TO ) = 0 for n > 0 (see [4, Section III, Corollary 10.2]). 3.3. Case 2
u Since ddx = (1 − ux)2 ddx , the G-module TO is isomorphic to O with the G-action given by  x  (1 − ux)2 (3.3.1) f u (x) = f 1 − ux for f ∈ O , u ∈ V . 3.4 The G-module O with G-action (3.3.1) is isomorphic to M ⊕ x O , where M = k ⊕ kx ⊕ kx 2 such that we have the following. (1) The G-module structure of M is given by LEMMA

(a0 + a1 x + a2 x 2 )u = a0 + (a1 − 2ua0 )x + (a2 − ua1 + u 2 a0 )x 2 ; that is, with respect to the basis {1, x, x 2 },  1 0  u ←→ 8(u) = −2u 1 u 2 −u (2)

 0 0 . 1

The G-module structure on the second factor x O is the original G-action on O ; that is, f u (x) = f (x/(1 − ux)) for f (x) ∈ x O .

Proof Note that O = M ⊕ x 3 O is a G-stable direct decomposition. The action of G on the first factor is as stated in (1). For x 2 f (x) ∈ x 3 O (i.e., f (x) ∈ x O ), the action (3.3.1) gives (x 2 f (x))u = x 2 f (x/(1 − ux)).

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

445

3.5 Let G act on x O as in Lemma 3.4(2), and let F = Frac(O ) be the fraction field of O . Since this action comes from that on O by ring automorphisms, it extends to F, respecting the decomposition F = x O ⊕ k[x −1 ]. We have H1 (G, x O ) ⊕ H1 (G, k[x −1 ]) = H1 (G, F) = 0 by standard facts (see [19, Section X.1]), and hence H1 (G, x O ) = 0. Thus H1 (G, TO ) ∼ = H1 (G, M). We now compute H1 (G, M). Since G is a commutative p-group, the condition for a map d : V → M to be a cocycle (d(u + v) = du + (dv)u ) implies ( d( pu) = 0 ⇐⇒ du + (du)u + (du)2u + · · · + (du)( p−1)u = 0, (i) d(u + v) = d(v + u) ⇐⇒ du + dv u = dv + du v .

(ii)

Let us write a cocycle d as du = a0 (u) + a1 (u)x + a2 (u)x 2 for u ∈ V . 3.5.1 Calculating the matrix 1 + 8(u) + 8(2u) + · · · + 8(( p − 1)u), we deduce that condition (i) is (a) empty unless p = 2 or 3, (b) equivalent to a0 (u) = 0 if p = 3, (c) equivalent to ua0 (u) + a1 (u) = 0 if p = 2. 3.5.2 Condition (ii) is equivalent to 2ua0 (u) = 2va0 (v) and u 2 a0 (v) − ua1 (v) = v 2 a0 (u) − va1 (u). Hence we have the following. (a) If p 6= 2, (ii) is equivalent to a0 (u) = ua0 and a1 (u) = u(a1 − ua0 ), where a0 and a1 are constants independent of u. (b) If p = 2, (ii) together with (i) is equivalent to a0 (u) = ua0 and a1 (u) = u 2 a0 , where a0 is a constant independent of u. Thus we get  2  ( p 6= 2, 3),  du = ua0 + u(a1 − ua 0 )x + a2 (u)x 2 Z1 (G, M) = du = ua1 x + a2 (u)x ( p = 3),   du = ua + u 2 a x + a (u)x 2 ( p = 2). 0 0 2 In each case, the cocycle condition is equivalent to the fact that the function a2 satisfies   a2 (u + v) = a2 (u) + a2 (v) + uv (u + v)a0 − a1

446

CORNELISSEN and KATO

(with a0 = 0 if p = 3 and a1 = 0 if p = 2). In particular, whenever a0 = a1 = 0, the function a2 is F p -linear. Hence Z1 (G, M) contains the t-dimensional k-subspace HomF p (V, k) = V ∗ ⊗ k. The dimension over k of Z1 (G, M) therefore is t + 2 if p 6= 2, 3 or is t + 1 otherwise. 3.6 Let g = b0 + b1 x + b2 x 2 . Then a coboundary is of the form g u − g = −2ub0 x + (−ub1 + u 2 b0 )x 2 . We can then compute a k-basis for H1 (G, TO ) as follows. 3.6.1 If p 6= 2, 3, a nontrivial such cohomology class, [d0 ], is given by the cocycle d0 with d0 u = −u + (u 2 + u)x −

1

1 1  u3 + u2 + u x 2. 3 2 6

The other cohomology classes come from the subspace {a0 = a1 = 0} ∼ = HomF p (V, k) in Z1 (G, M), in which the coboundary classes are spanned by du = ux 2 . Hence the part of the cohomology coming from {a0 = a1 = 0} is isomorphic to HomF p (V, k)/k · ι, where ι : V ,→ k is the natural inclusion. Hence H1 (G, TO ) ∼ = k · [d0 ] ⊕ HomF p (V, k)/k · ι. In particular, dimk H1 (G, TO ) = t. 3.6.2 If p = 3, then only the cocycles coming from {a0 = a1 = 0} survive. Hence H1 (G, TO ) ∼ = HomF3 (V, k)/k · ι, and we have dimk H1 (G, TO ) = t − 1. 3.6.3 If p = 2, we use an ad hoc construction. Let {u 1 , . . . , u t } be a basis of V , define a cocycle de0 by de0 (u i ) := u i −u i2 x on basis elements, and use de0 (u+v) := de0 u+(de0 v)u inductively to define de0 u for all u. Notice that this requirement is compatible with conditions (i) (de0 (2u) = 0) and (ii) (commutativity) from Section 3.5. Thus we get a well-defined cocycle on all of V since p = 2.

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

447

The remaining part, {a0 = a1 = 0} ∼ = HomF2 (V, k), contains two-dimensional coboundaries spanned by ι and the Frobenius embedding Frob : V ,→ k : u 7→ u 2 . Hence H1 (G, TO ) ∼ = k · [de0 ] ⊕ HomF2 (V, k)/(k · ι + k · Frob), and, as the sum k · ι + k · Frob is direct precisely when t > 1, we get that dimk H1 (G, TO ) is t − 1 if t > 1 or is 1 if t = 1. 3.7. Case 3 Since |H | is coprime to the characteristic of k, we have H1 (G, M) = H1 (N , M) H

(3.7.1)

(see, e.g., [4, Section III, Proposition 10.4]). The calculation of H1 (N , M) is similar to the one in Sections 3.5 and 3.6, but F p is replaced by Fq (for q = p s ) everywhere. The only difference is for p = 2 because then the subspace HomFq (V, k) in HomF p (V, k) has a trivial intersection with the one-dimensional subspace spanned by the Frobenius embedding u 7→ u 2 . We now describe the action of H on the cohomology H1 (N , M). First, recall that the action of H on TO is given by  d τ d xr = ζ r −1 x r . dx dx It stabilizes M, on which it acts by (a0 + a1 x + a2 x 2 )τ = ζ −1 a0 + a1 x + ζ a2 x 2 . The action of H on the cohomology H1 (N , M) is induced from the action on the space −1 of cocycles given by d τ u = (du τ )τ (cf. [4, Section III.8]). Hence if p 6= 2, we get −1 d0τ u = (d0 ζ u)τ = ζ 2 d0 u + (ζ − ζ 2 )ux − ((1/2)(ζ − ζ 2 )u 2 + (1/6)(1 − ζ 2 )u)x 2 , where the last two terms form a coboundary; thus d0τ = ζ 2 d0 , and if p = 2, a similar result holds for the cocyle de0 introduced in Section 3.6.3. This implies that the class d0 is not H -invariant as long as n 6= 2, and de0 is not H -invariant at all. Next we look at the remaining part {a0 = a1 = 0} ∼ = HomFq (V, k). An element a2 ∈ HomFq (V, k) (corresponding to the cocycle du = a2 (u)x 2 ) is H -invariant if and only if d τ u = ζ −1 a2 (ζ u)x 2 = a2 (u)x 2 or, equivalently, a2 (ζ u) = ζ a2 (u). Since ζ generates Fq , it is equivalent to the Fq -linearity of a2 . Summing up, we have the following.

448

CORNELISSEN and KATO

3.7.2 Suppose p 6= 2, 3. If n 6= 2, then H1 (G, TO ) ∼ = H1 (N , M) H ∼ = HomFq (V, k)/k · ι and hence is of dimension t/s − 1. If n = 2 (which implies s = 1), it is of dimension t since d0 is also H -invariant. 3.7.3 If p = 3, then we always have H1 (G, TO ) ∼ = H1 (N , M) H ∼ = HomFq (V, k)/k · ι, and dimk H1 (G, TO ) = t/s − 1. 3.7.4 If p = 2, then n 6= 2 and s > 1 (since 1 6= n|2s − 1). Hence H1 (G, TO ) ∼ = H1 (N , M) H ∼ = HomFq (V, k)/k · ι (recall that the Frobenius embedding is no longer present), and dimk H1 (G, TO ) = t/s − 1. Remark 3.8 Fix a subgroup V1 = (Z/ pZ) of V , and consider the inflation restriction sequence for this subgroup (with V 0 := V /V1 ): inf

res

0

0 → H 1 (V 0 , M V1 ) −→ H 1 (V, M) −→ H 1 (V1 , M)V . The left cohomology group is the following (for p 6= 2): The invariants M V1 are just kx 2 , on which V 0 acts trivially; so the group is isomorphic to Hom(V 0 , k), which maps via inflation to the part HomF p (V, k)/k · ι of the cohomology H 1 (G, TO ). On the other hand, the part [d0 ] is precisely the one that is nonzero when mapped under restriction to H 1 (V1 , M) = H 1 (Z/ pZ, TO ). This group is the one studied in [1] (see, e.g., [1, Lemme 4.2.2]), where an intrinsic characterization of the class [d0 ] is provided. 4. The local prorepresentable hull In this section we calculate the hull Hρ of Dρ , where the group G and its action on R = O are as in Cases 1, 2, and 3. The first one is a trivial case; that is, the tame cyclic action is rigid (cf. [1, Section 4.3]).

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

449

To analyze Case 2, we have to look at lifting obstructions, much as is the case in [1, Section 4.2]. In our case, liftings are related to what we call formal truncated Chebyshev polynomials, which we now introduce. 4.1. Formal truncated Chebyshev polynomials For a positive integer N , we set P N −1 u+k
k  P N u+k−1
k  α α α k=0 [N ] k=0 2k+1 2k
k
P N u+k , Mα,β (u) = P N −1 u+k k α α + β(u) k=0 2k k=0 2k+1 where u, α, and β(u) are indeterminates. The entries are considered to be formal power series of these indeterminates with coefficients in Q. (This means in particular that the binomial coefficients evaluate to polynomials in u.) We denote by Mα[N ] (u) [N ] (1 ≤ N ≤ ∞) the matrix Mα,β (u) with β(u) replaced by 0. As a matrix of formal power series, we have an identity   Su (1 + α2 ) − (1 + α)Su−1 (1 + α2 ) αSu−1 (1 + α2 ) Mα[∞] (u) = , Su−1 (1 + α2 ) Su (1 + α2 ) − Su−1 (1 + α2 ) where Su (x) is the Chebyshev polynomial of second kind,  x 3 Su (x) = (u + 1)2 F1 − u, u + 2, ; − . 2 4 Recall that if u ∈ Z≥0 , then Su (x) is defined by the generating series ∞

X 1 = Su (x)r u 1 − 2xr + r 2 u=0

or by sin(u + 1)θ . sin θ The above identity of matrices follows from the following easy fact: Su (x) =

 ∞   x X u +k +1 Su 1 + = xk. 2k + 1 2 k=0

For 1 ≤ N ≤ ∞, let us write Mα[N ] (u)

 =

] A[N α (u) Cα[N ] (u)

Bα[N ] (u) Dα[N ] (u)



.

Then it follows easily from the basic recursion between binomial coefficients that Bα[N ] (u) = αCα[N ] (u)

and

] [N ] [N ] A[N α (u) + Bα (u) = Dα (u);

450

CORNELISSEN and KATO

that is, the relations between the coefficients that hold for N = ∞ are also true for the truncated versions. It follows formally from these identities between entries that Mα[N ] (u)Mα[N ] (v) = Mα[N ] (v)Mα[N ] (u)

(4.1.1)

as identities of matrices of formal power series in u, v, and α. Next, we observe the following trigonometric identities: (i) Su+v (x) + Su−1 (x)Sv−1 (x) = Su (x)Sv (x), (ii) Su+v−1 (x) + 2x Su−1 (x)Sv−1 (x) = Su−1 (x)Sv (x) + Su (x)Sv−1 (x), (iii) Su (x)2 − 2x Su (x)Su−1 (x) + Su−1 (x)2 = 1. The first two of these imply Mα[∞] (u)Mα[∞] (v) = Mα[∞] (u + v),

(4.1.2)

and the third one implies det Mα[∞] (u) = 1.

(4.1.3)

4.2. Lifting obstructions Now assume p 6= 2, and let A be an artinian local k algebra with A/m A = k. Let α ∈ m A . Let V ⊂ k be a finite-dimensional F p -vector space, and let β : V → m A be an F p -linear map. For u ∈ V ,→ k we use the notation eα,β (u) = M [( p−1)/2] (u) M α,β

and

eα (u) = Mα[( p−1)/2] (u) M

(where multiplication of u’s takes place inside k). By (4.1.1), we have eα (u) M eα (v) = M eα (v) M eα (u). M

(4.2.1)

Also, by (4.1.2) and (4.1.3), we get

and

eα (u) M eα (v) ≡ M eα (u + v) mod (α ( p−1)/2 ) M

(4.2.2)

eα (u) ≡ 1 mod (α ( p−1)/2+1 ). det M

(4.2.3)

We now look at what happens if β 6≡ 0. A small calculation shows that the commueα,β (u) M eα,β (v) = M eα,β (v) M eα,β (u) is equivalent to αβ(u)C(v) = tation relation M αβ(v)C(u) (where C = Cα depends on α). Putting v = 1 (where we tacitly assume to have conjugated the action of V such that F p ⊆ V (see [1, Lemme 4.2.1])), we get αβ(u)C(1) = αβ(1)C(u). Thus αβ(1)C(u) is a linear map in u. Looking at the constant term using the explicit
form of C(u)(= 1 + u+1 2 α + · · · ), we get αβ(1) = 0. If we substitute this back into

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

451

the previous display, we find αβ(u)C(1) = 0, but since C(1) = 1 + α is invertible in A (recall that α ∈ m A ), we finally get αβ(u) = 0 for all u. Conversely, if αβ = 0, the commutation relation clearly holds. Hence eα,β (u) M eα,β (v) = M eα,β (v) M eα,β (u) ⇐⇒ αβ = 0. M

(4.2.4)

Looking at (4.2.3), we get the following. eα,β (u) ≡ 1 mod (α ( p+1)/2 ). If αβ = 0, then det M

(4.2.5)

LEMMA 4.2.6 The following conditions are equivalent: (i) the formula

x u :=

ax + b , cx + d

where

  ab eα,β (−u), =M cd

(4.2.6.1)

defines a lift of the action of V on TO to A; eα,β (u) M eα,β (v) = γ M eα,β (u + v) for all u, v ∈ V and a γ ∈ A (possibly M depending on u, v, α, β); (iii) α ( p−1)/2 = αβ = 0. Actually, if this holds, we have γ = 1. (ii)

Proof Looking at the relations in V , we see the equivalence of (i) and (ii). (ii) =⇒ (iii). The relation αβ = 0 follows from (4.2.4). Since the identity in (i) obviously holds with γ = 1 if A = k, we can set γ = 1 + δ with δ ∈ m A . Taking determinants, it follows from (4.2.3) that γ 2 (= 1 + δ(2 + δ)) = 1 + Pα ( p+1)/2 for some P ∈ A. Since δ + 2 is invertible in A, γ ≡ 1 mod α N +1

(4.2.6.2)

for N = ( p − 1)/2. The coefficient of α ( p−1)/2 in the lower-left entry of eα,β (u) M eα,β (v) is M N −1  X k=0

  NX   −1  v+ N −k−1 v+k u+ N −k + . 2(N − k) 2k + 1 2(N − k)

u+k 2k + 1

(4.2.6.3)

k=0

(Note that we have used the fact that we know already that αβ = 0; so there is no contribution from β in this calculation.) We now put u = N and v = 2 (again assuming tacitly that F p ⊆ V ) into (4.2.6.3). The result is 1, so the above coefficient (4.2.6.3) eα,β (u + v) is is nonzero. Since the coefficient of α ( p−1)/2 in the lower-left entry of M N zero, we conclude from this and (4.2.6.2) that α = 0, as desired.

452

CORNELISSEN and KATO

(iii) =⇒ (ii). If β ≡ 0, this is already in (4.2.2); so let β ≡ 6 0. Using αβ = 0 and eα,β (u) M eα,β (v) as A(u) ≡ 1 mod α, we calculate the entries of the matrix M := M follows: M1,1 = A(u)A(v) + αC(u)C(v), M1,2 = α A(u)C(v) + α A(v)C(u) + α 2 C(u)C(v), M2,1 = A(v)C(u) + A(u)C(v) + αC(u)C(v) + β(u + v), M2,2 = A(u)A(v) + α A(u)C(v) + α A(v)C(u) + α(1 + α)C(u)C(v). Note that, except for the third one, these entries are independent of β. Since we already have (4.2.2), we find that these entries are equal to A(u + v), αC(u + v), C(u + v) + β(u + v), and A(u + v) + αC(u + v), respectively. Thus we have the relation in (i) with γ = 1. Remarks 4.2.6.4 Notice the sign change from u to −u in (i) since we are lifting x u = x/(1 − x). We see that the condition α ( p−1)/2 = 0 corresponds to the order- p condition on the lift (as in [1]), whereas αβ = 0 gives the obstruction to lifting the commutativity of V . 4.3. Explicit infinitesimal liftings We continue for the time being to assume that p 6= 2. At the infinitesimal level A = k[] with  2 = 0, we let α = a0  and β(u) = − · φ(u) for a linear map φ ∈ HomF p (V, k). Lemma 4.2.6 assures us of the fact that (4.2.6.1) does define a first-order lifting as long as α = 0 for p = 3. It is explicitly given as xu =

(1 + (1/2)u(u + 1)a0 )x − ua0  . 1 − (u + (1/6)u(u 2 − 1)a0  − φ(u))x + (1/2)u(u − 1)a0 

By the formula in Proposition 2.3, the corresponding cocycle d is given as d  x u  du = d 1 + ux u =0    1 1 1  = −a0 u + a0 (u 2 + u)x − a0 u 3 + u 2 + u − φ(u) x 2 3 2 6 = a0 d0 + φ in Z 1 (V, M) = kd0 + HomF p (V, k), where d0 is as in Section 3.6.1. This means that (4.2.6.1) for α = 0 defines a lifting in the direction of φ (which is unobstructed) and for β = 0 in the direction of a0 [d0 ] (obstructed by α ( p−1)/2 = 0). If both α and β are nonzero, a lifting in the direction of a0 [d0 ] + φ is obstructed by the equations in Lemma 4.2.6.

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

453

4.4. Calculating the hull 4.4.1. The case p 6= 2, 3 If we let R be the ring ( p−1)/2

R = k[[x0 , x1 , . . . , xt ]]/hx0

, x0 x1 , x0 x2 , . . . , x0 xt , x1 + · · · + xt i,

then Lemma 4.2.6 shows that (4.2.6.1) defines a lifting of ρ to R, and hence there is a morphism of functors Hom(R, −) → Dρ . To prove that R is actually the hull Hρ of Dρ , we argue as in [1, p. 217]. It suffices to prove that R is a versal deformation, that is, that the above morphism is smooth and is an isomorphism on the level of tangent spaces. The latter is clear from our computation of the tangent space to Dρ from group cohomology and the above explicit form of R. To prove the former, let A0 → A be a small extension in Ck with kernel I . We have to show that Hom(R, A0 ) → D(A0 ) × D(A) Hom(R, A) is surjective. So assume that ρ ∼ ∈ im(Hom(R, A) → D(A)) lifts to ρ ≈ in D(A0 ). This means the corresponding obstruction in H2 (G, TO ) ⊗ I is zero. Since G = (Z/ pZ)t , this obstruction measures exactly the possible failure of the commutation relation ρ ≈ (u + v) = ρ ≈ (u)ρ ≈ (v), ∀u, v ∈ V , which we know by Lemma 4.2.6 is eα 0 ,β 0 (u) to Hom(R, A0 ), and we can given by the equations in R. So we can lift via M adjust this lifting in such a way that its image in D(A0 ) coincides with the given one since the tangent spaces to the two functors are isomorphic. (Note that the fibers are H 1 (G, TO ) ⊗ I -torsors.) This finishes the proof of smoothness. 4.4.2. The case p = 3 The same argument works, except that the class [d0 ] does not occur so that all liftings in the direction of HomF p (V, k) are unobstructed. The result is Hρ = k[[x1 , . . . , xt ]]/hx1 + · · · + xt i. 4.4.3. The case p = 2 We deal with this case by an ad hoc construction. Recall that in this case G ∼ =V ⊆k is a t-dimensional F2 -vector space for which we pick a basis {u 1 , . . . , u t }. As before, let α ∈ A, and let β : V → m A be a linear map. We lift the action of the basis u i by x u i :=

x + αu i . (u i + β(u i ))x + 1

We have (x u i )u i = x, and we observe that (x u i )u j = (x u j )u i is equivalent to αu i β(u j ) = αu j β(u i )

(4.4.4)

454

CORNELISSEN and KATO

for all i, j. We now define a lift to any element of V by x u i1 +···+u il := x u i1 ◦···◦u il for i 1 , . . . , il ∈ {1, . . . , t}. This forces commutativity to hold. Notice that this only gives a well-defined lift to all of V since p = 2. If we set A = K [], α = a0 , β = φ(u), for a linear map φ ∈ HomF p (V, k), then we find that this lift corresponds to the cocycle a0 d0 + φ ∈ Z 1 (V, M), so that we are indeed lifting in all directions of the tangent space. One can now reason as before to find that the hull of Dρ is given by Hρ = k[[x0 , x1 , . . . , xt ]]/hx1 + · · · + xt , u 1 x1 + · · · + u t xt , x0 (xi u j − x j u i )i, j=1,...,t i. In Figure 1, one sees a schematic representation of the geometrical structure of these hulls. For p 6= 2, a nilpotent zero-dimensional scheme is attached to a polydisc in the [d0 ]-tangential direction. Hom(V, k)directions

Hom(V, k)directions



 



  p 6= 2

[d0 ]-direction

  [d0 ]-direction   p=2

Figure 1. Pictorial representation of the local versal deformation ring

4.4.5. The case n 6= 1 Finally, in Case 3 of Section 3.1, the arguments are completely similar to those of the preceding paragraphs; the results are as follows. (1) If n 6= 2 or p = 2, 3, then only first-order deformations coming from HomFq (V, k) occur, and these can be lifted without obstruction: Hρ ∼ = k[[x1 , . . . , xt/s ]]/hx1 + · · · + xt/s i. (2)

If p 6= 2, 3 and n = 2 (hence s = 1), then the action of H extends to the lifting (4.2.6.1) (just replacing u by ζ u) and the obstructions do not change. Hence we have ( p−1)/2 Hρ ∼ , x0 x1 , . . . , x0 xt , x1 + · · · + xt i. = k[[x0 , x1 , . . . , xt ]]/hx0

THEOREM 4.5 Let ρ : G → Aut(TO ) be a local representation of a finite group G, where O is of 0 characteristic p. Let n be an integer coprime to p, define s := min{s 0 : n| p s − 1}, and let [d0 ] be the cohomology class defined in Section 3.6. Table 1 lists the dimension

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

455

of the group cohomology H 1 (G, TO ); whether [d0 ] is trivial (—), unobstructed (unobs.), or leads to obstructions (obs.); and the Krull dimension dim Hρ of the prorepresentable hull Hρ of the local deformation functor Dρ .

Table 1

G Z/n (Z/ p)t

(Z/ p)t o Z/n

( p, t, n) p 6= 2, 3 p=3 p = 2, t > 1 p = 2, t = 1 n 6= 2 or p = 2, 3 n=2

h 1 (G, TO ) 0 t t −1 t −1 1 t/s − 1 t

[d0 ] — obs. — obs. unobs. — obs.

dim Hρ 0 t −1 t −1 t −2 1 t/s − 1 t −1

Remark 4.5.1 For n = t = 1, this result agrees with [1, Proposition 4.1.1], where it is shown that h 1 (G, TO ) =

j 2β k p P∞

−

lβ m p

for G = Z/ p a cyclic p-group and β = j=0 (|G i | − 1). (Recall that G i are the higher ramification groups.) Indeed, in our case, G 0 = G 1 = G and G i = 0 for i > 1, so that β = 2 p − 2 and we get    3 − 2 = 1 if p > 3, 1 h (G, TO ) = 2 − 2 = 0 if p = 3,   2 − 1 = 1 if p = 2. Similarly, our calculation of the hull and its Krull dimension is compatible with the results from Bertin and M´ezard [1] if we observe that (in their notation, cf. [1, p. 215]) ψ(X ) = X ( p−1)/2 mod p ( p > 2). Remark 4.5.2 Obstructions to lifting first-order deformations are certain second cohomology classes in H2 (G, TO ), but these can form a strict subset of this cohomology group. If t = 1, then the group is easy to calculate directly (cf. [1]) or is seen to be isomorphic to H1 (G, TO ) by Herbrand’s theorem since G is cyclic. This already shows that for t = 1, obstructions can form only a small part of H2 (G, TO ). The computation of

456

CORNELISSEN and KATO

this second cohomology group in the general case t > 1 (so G is no longer cyclic) seems rather tedious. 5. Main theorem on algebraic equivariant deformation We have now collected all information needed to prove our main algebraic theorem. THEOREM 5.1 Let X be an ordinary curve over a field of characteristic p > 0, and let G be a finite group acting on X via ρ : G → Aut(X ). (a) The Krull dimension of the prorepresentable hull of the deformation functor D X,ρ is given by s X dim H X,ρ = 3gY − 3 + δ + dim Hρi , i=1

where gY is the genus of Y := G\X , δ is given in Notation 1.5, and Hρi is the prorepresentable hull of the local deformation functor associated to the representation of the ramification group G i at the branch point wi in X → Y , whose dimension was given in Theorem 4.5, except in the following four cases: (1) when p = 2, Y = P1 , and X → Y is branched above two points, then dim H X,ρ = dim Hρ1 + dim Hρ2 ; (2) when X = P1 → Y = P1 is tamely branched above two points, then dim H X,ρ = 0; (3) when X = P1 → Y = P1 is wildly branched above a unique point, then dim H X,ρ = dim Hρ1 (in this case, G is a pure- p group and X → Y is an Artin-Schreier cover); (4) when X → Y is an unramified cover of elliptic curves, then dim H X,ρ = 1. Furthermore, if the genus g of X is greater than or equal to 2, then D X,ρ is prorepresentable by H X,ρ . (b) The dimension of the tangent space to the functor D X,ρ as a k-vector space satisfies   if p 6= 2, 3,  #{i : n i ≤ 2} dimk D X,ρ (k[]) = dim H X,ρ +

0   #{i : n = 1 and t > 1} i i

Proof Let Y be the quotient G\X . From Proposition 1.12 we find that dim H X,ρ =

s X i=1

dim Hρi + h 1 (Y, π∗G T X ),

if p = 3,

if p = 2.

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

457

where Hρi are the hulls of the local deformation functors associated to the action of the ramification groups G i at wild ramifications points w1 , . . . , ws on the space of local derivations, which was computed in Theorem 4.5. To compute the h 1 -term, recall from Proposition 1.6 that π∗G T X = TY (−1), where 1 is defined in Notation 1.5. By the Riemann-Roch theorem, we find that
h 1 (π∗G T X ) = 3gY − 3 + δ + h 0 TY (−1) , where gY is the genus of Y . Since deg(TY (−1)) = 2 − 2gY − δ, the last term vanishes if gY > 1 or gY = 1 and δ > 0 or gY = 0 and δ > 2. If gY = 1 and δ = 0, X is an unramified cover of an elliptic curve and hence is an elliptic curve itself. Assume that gY = 0, that there are at least two branch points on Y , and that δ ≤ 2. If p 6= 2, then these branch points have to be tame, so δ = 2, and the Hurwitz formula implies that g X = 0 too. If p = 2, they can be wild, but both ramification groups have to be Z/2Z, so we still have δ = 2. In both cases, h 0 (TY (−δ)) = h 0 (OP1 ) = 1. If gY = 0 and only one point on Y is branched, then it follows from Hurwitz’s formula (using the fact that second ramification groups vanish in the ordinary case; cf. [17]) that g X = 0 too, and the ramification has to be wild at this point. So if p 6= 2 or p = 2, t > 1, we find that δ + h 0 (TY (−δ)) = 2 + h 0 (OP1 ) = 3. On the other hand, if p = 2 and t = 1, then δ + h 0 (TY (−δ)) = 1 + h 0 (OP1 (1)) = 3. Let np t (with n coprime to p) be the order of the ramification group at that unique point. Hurwitz’s formula gives in particular that (n − 1) p t + 2 divides 2np t , and this (together with n| p t − 1) implies n = 1. Hence we do get a (Z/ pZ)-cover. This finishes the proof of part (a). For part (b), we apply the formula from Proposition 1.9 in combination with (1.7.1). It thus suffices to compute h 0 (Y, H 1 (G, T X )), but H 1 (G, T X ) is concentrated in the branch points wi , where it equals the group cohomology H 1 (G i , TOwi ) (see [1, Section 3.3]), so dim H X,ρ and dimk D X,ρ (k[]) differ only at places where [d0 ] is obstructed in Table 1. Remark 5.1.1 The main algebraic theorem stated in the introduction follows from Theorem 5.1 by excluding the cases g ≤ 2, p = 2, 3. Example 5.1.2 (Artin-Schreier curves) t t The Artin-Schreier curve whose affine equation is given by (y p − y)(x p − x) = c for some constant c ∈ k ∗ has automorphism group G = (Z/ pZ)2t o D pt −1 ,

458

CORNELISSEN and KATO

where D∗ denotes a dihedral group of order 2∗. The quotient Y = G\X is a projective line, and the branching groups are Z/2Z (twice if p 6= 2 and once if p = 2) and (Z/ pZ)t o Z/( p t − 1)Z (once). This curve with its full automorphism groups hence allows for a one-dimensional deformation space (for different reasons if p 6= 2 and p = 2). This deformation is given exactly by varying c. Example 5.1.3 (Drinfeld modular curves) The Drinfeld modular curves X (n) from the introduction have automorphism group G := 0(1)/ 0(n)Fq∗ for d := deg(n) > 1. The quotient Y := G\X (n) is a projective line, over which X is branched at two points with ramification groups Z/( p + 1)Z and Fqd o Fq∗ , respectively. Hence X (n) can be deformed in d − 1 ways (regardless of p, but for different reasons if p = 2—in that case we are in exceptional case (1) from Theorem 5.1). Part B: Analytic theory 6. Equivariant deformation of Mumford curves 6.1. Mumford curves Let (K , | · |) be a complete discrete valuation field with valuation ring O K and residue field O K /mO K = k. Recall that a projective curve X over K is called a Mumford curve if it is uniformized over K by a Schottky group. This means that there exists a free subgroup 0 in PGL(2, K ) of rank g, acting on P1K with limit set L0 such that X 1,an satisfies X an ∼ = 0\(P K − L0 ) as rigid analytic spaces. Mumford [16] has shown that these conditions are equivalent to the existence of a stable model of X over O K whose special fiber consists only of rational components with k-rational double points. Because of the GAGA-correspondence for one-dimensional rigid analytic spaces, we do not have to (and do not) distinguish between analytic and algebraic curves. It is well known that Mumford curves are ordinary. (This is basically because their Jacobian is uniformized by (Ganm,K )g / 0 ab , where g is the genus of X ; cf. [5, Lemma 1.2].) Thus the results from the previous section essentially solve the equivariant deformation problem for Mumford curves in a cohomological way. In this part, however, we want to develop an independent theory of analytic deformation of Mumford curves based on the groups that uniformize them. This makes the liftings and obstructions whose cohomological existence was proven in the previous part more visible as actual deformations of (2 × 2)-matrices over K .

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

459

6.2. Automorphisms It is well known (see [5, Theorem 1.3]) that for a Mumford curve X of genus g ≥ 2 with Schottky group 0, Aut(X ) = N (0)/ 0, where N (0) is the normalizer of 0 in PGL(2, K ). Conversely, if N is a discrete subgroup of PGL(2, K ) containing 0 and contained in N (0), then it induces a group of automorphisms ρ : N / 0 ,→ Aut(X ). Notation 6.3 If a finitely generated discrete subgroup N of PGL(2, K ) is given, let Hom∗ (N , PGL(2, K )) denote the set of injective homomorphisms φ : N → PGL(2, K ) with discrete image. Then such an N contains a finite-index normal free subgroup of finite rank 0 (see [9, Section I.3]), and if 0 is nontrivial, the pair (N , 0) gives rise to a Mumford curve with an action of N / 0, as is being considered here. 6.4. Rigidity Two Mumford curves X, X 0 with Schottky groups 0, 0 0 are isomorphic if and only if 0 and 0 0 are conjugate in PGL(2, K ) (see [16, Corollary 4.11]). Remark 6.4.1 Note that this is very different from the situation in the uniformization theory of Riemann surfaces S, where, in a representation S = 0\ with 0 a Schottky subgroup of PGL(2, C), the domain of discontinuity  of 0 is not the universal topological covering space of S; this does hold for Mumford curves. 6.5. Analytic deformation functors Recall that C K is the category of local Artinian K -algebras. If N and φ ∈ Hom∗ (N , PGL(2, K )) are given, we consider the analytic deformation functor D N ,φ : C K → Set of the pair (N , φ), which sends A ∈ C K to the set of liftings of (N , φ) to A. Here, a lifting is a morphism φ ∼ ∈ Hom(N , PGL(2, A)) which, when composed with reduction modulo the maximal ideal m A of A, equals the original embedding φ. (In particular, φ ∼ is injective.) Note that we do not consider classes of liftings modulo conjugacy by PGL(2)—this implies that D N ,φ is naturally equipped with an action of group functor PGL(2)∧ given by   PGL(2)∧ : C K → Groups : A 7→ ker PGL(2, A) → PGL(2, K ) , and we denote the quotient by ∧ D∼ N ,φ := PGL(2) \D N ,φ .

460

CORNELISSEN and KATO

Since N is finitely generated, it is not difficult to show that the functors D N ,φ and ∼ D∼ N ,φ have prorepresentable hulls H N ,φ and H N ,φ . We want to compute the dimension of the tangent spaces to these functors, as well as the Krull dimension of the hulls H N ,φ and H N∼,φ , and this is done by decomposing N using its structure as a group acting on a tree to give a decomposition of D N ,φ . Note that such a decomposition is not given on the level of D ∼ N ,φ . 7. Structure of N as a group acting on a tree We fix a finitely generated discrete subgroup N of PGL(2, K ), and we now recall how the structure of N can be seen from its action on the Bruhat-Tits tree (cf. [5, Section 2]). 7.1. The Bruhat-Tits tree Let T denote the Bruhat-Tits tree of PGL(2, K ) (i.e., its vertices are similarity classes 3 of rank two O K -lattices in K 2 , and two vertices are connected by an edge if the corresponding quotient module has length one; see Serre [20] and L. Gerritzen and M. van der Put [9]). We assume K to be large enough so that all fixed points of N are defined over K ; then N acts without inversion on T . It is a regular tree in which the edges emanating from a given vertex are in one-to-one correspondence with P1 (k). The tree T admits a left action by PGL(2, K ). 7.2. Notation on trees For any subtree T of T , let Ends(T ) denote its set of ends (i.e., equivalence classes of half-lines differing by a finite segment). There is a natural correspondence between P1 (K ) and Ends(T ). Let V (T ) and E(T ) denote, respectively, the set of vertices and edges of T . For σ ∈ E(T ), let o(σ ) (resp., t (σ )) denote the origin (resp., terminal) vertex of σ . Let N x denote the stabilizer of a vertex or edge x of T for the action of N . The maps o, t induce maps Nσ → N3 for 3 = o(σ ) or 3 = t (σ ), which are denoted by the same letter. For any u, v ∈ P1 (K ), let ]u, v[ denote the apartment in T connecting u and v (seen as ends of T ). 7.3. The trees associated to N We can construct a locally finite tree T (L ) (possibly empty) from any compact subset L of P1 (K ); it is the minimal subtree of T whose set of ends coincides with L S or, equivalently, the minimal subtree of T containing u,v∈L ]u, v[. We define T N to be the tree associated to the subset L N consisting of the limit points of N in P1 (K ). Since N is a finitely generated discrete group, T N coincides with the tree of N as it is defined in Gerritzen and van der Put [9].

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

461

7.4. The graph associated to N T N admits a natural action of N , and we denote the quotient graph by TN := N \T N ; the corresponding quotient map is denoted by π N . The graph TN is finite and connected. We turn TN into a graph of groups as follows. Let T be a spanning tree (maximal subtree) of TN , which we can see as a subtree of T N by a fixed section ι : T → T N of π N . Let c = c(TN ) denote the cyclomatic number of TN (equal to the number of edges outside T ), and fix 2c lifts ei± of these edges outside T to T N which satisfy c t (ei+ ) ∈ V (ι(T )), o(ei− ) ∈ V (ι(T )). Fix c hyperbolic elements {γi }i=1 in N such that + − + c γi ei = ei . Then ι(T ) ∪ {ei }i=1 is a fundamental domain for the action of N on T N . For any vertex 3 ∈ V (T N ) and edge σ = [3, M] ∈ E(T N ), we denote by N3 and Nσ = N3 ∩ N M their respective stabilizers for the action of N . Note that these groups are finite since N is discrete. For a vertex v ∈ V (TN ) = V (T ), we let Nv = Nι(v) . For edges e ∈ E(TN ), either e ∈ E(T ) and we let Ne = Nι(e) , or else there is a unique i such that π N (ei± ) = e and we let Ne = Ne+ . i The morphisms between these groups are defined as follows: If e ∈ E(T ), then Ne ,→ Nt (e) and Ne ,→ No(e) are the natural inclusions; if, on the other hand, e = π N (ei± ), then Ne ,→ Nt (e) is the natural inclusion but Ne ,→ No(e) is given by s 7→ γi−1 sγi . We then have the following description of the group N . THEOREM 7.5 (Bass-Serre theorem; see [6, Theorem I.4.1, Proposition I.4.4]; cf. [20]) For any spanning tree T of TN , N equals the fundamental group of the graph of groups TN at T . This means that N is generated by the amalgam of Nv over Ne for all e ∈ E(TN ), v ∈ V (TN ) together with the fundamental group of TN at T as a plain c , where c = c(T ) is the graph, namely, the free group Fc on c generators {n i }i=1 N cyclomatic number of TN . The further relations in N are of the form n i t (γ )n i−1 = o(γ ) for every i = 1, . . . , c and for every γ ∈ Ne , e ∈ TN − T . In particular, there is a split exact sequence of groups

0 → lim N•∼ → N →Fc(TN ) → 0, −→

TN∼

where π : TN∼ → TN is the universal covering of TN as a plain graph, which has been made into a graph of groups by setting N•∼ for • ∈ V (TN∼ ) ∪ E(TN∼ ) equal to Nπ(•) .

462

CORNELISSEN and KATO

8. Decomposition of the functor D N ,φ 8.1 Let s : Fc(TN ) → N be a splitting of the sequence in Theorem 7.5. Then there is an isomorphism of functors PROPOSITION

D N ,φ ∼ D N• ,φ| N• × D Fc ,φ◦s , = lim ←− TN

where the inverse limit is in the category of functors. (Note that morphisms between N• naturally induce morphisms of functors between D N• .) Remark 8.1.1 Note that we get a direct product of functors, but a limit of functors over TN (instead of the obvious semidirect product and limit over TN∼ ). We also note that there is no such decomposition on the level of the functors D ∼ N ,φ . Proof Let A ∈ C K . By restriction, a deformation of N to A trivially gives rise to deformations of N• and Fc . For the rest of the proof, we imitate the construction of TN as a graph of groups, but we lift to TN∼ instead of T N . So choose a fixed maximal spanning tree ι : T ,→ TN∼ and a basis {γ1 , . . . , γc } of s(Fc ), where c = c(TN ). Take, as before, 2c edges ei± ∈ E(TN∼ ) such that t (ei+ ) ∈ V (T ), o(ei− ) ∈ V (T ), γi ei+ = ei− . Thus T ∪ {ei+ } is a fundamental domain for TN∼ → TN . To give elements in lim D N• ,φ| N• (A) ←− TN

and

D Fc ,φ◦s (A)

means precisely to give a compatible collection of φv : Nv ,→ PGL(2, A) and φc : Fc ,→ PGL(2, A). Compatibility means that for e ∈ E(TN ), the following diagram is commutative: / No(e) Ne 

Nt (e)

φt (e)

 φo(e) / PGL(2, A)

We want to extend this to an embedding of N . By the construction of the fundamental domain, there exists for any v ∈ V (TN∼ ) a unique γ ∈ s(Fc ) such that v ∈ γ T , and this allows us to define Nv → PGL(2, A) to be σ 7→ φγ −1 v (γ −1 σ γ ). For edges e, we similarly get γ such that e ∈ γ · (T ∪ c ), and the same works. {ei+ }i=1

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

463

By the compatibility of φv and φc expressed in the commutative diagram above, we thus get an embedding of lim N• into PGL(2, A), −→

TN∼

which by construction is compatible with the conjugation action of Fc , so that we finally get an embedding N ,→ PGL(2, A), namely, an element of D N ,φ (A). Since this construction is functorial in A, we get the desired inverse map of functors. 8.2. Computing the functor D Fc ,φ◦s The set of morphisms
Hom Fc , PGL(2, K ) is a smooth algebraic variety over K ; by choosing a basis of Fc , it is isomorphic to PGL(2, K )c over K . We can take its formal completion at the K -rational point φ ◦ s and then
∧ D Fc ,φ◦s ∼ = Hom Fc , PGL(2, K ) φ◦s as formal functors. In particular,
dim K D Fc ,φ◦s K [] = dim H Fc ,φ◦s = 3c, where the first expression is the dimension of the tangent space and the second expression is the Krull dimension of the prorepresentable hull of the functor. 8.3. Computing the functor D N ,φ for finite N Here the argument is based on the simple observation that an injective element φ of Hom(N , PGL(2, K )) corresponds to a cover P1 → P1 with Galois group N ; hence it is related to the algebraic deformation functor (Section 1.1) of the pair (P1 , φ) (regarding φ as a representation of N into Aut(P1 )). The functor DP1 ,φ is defined modulo conjugation by PGL(2), whereas the analytic deformation functor D N ,φ carries a natural action of PGL(2)∧ . However, it is easy to see that D∼ N ,φ = DP1 ,φ . From this formula we get in particular that
dim H N ,φ = dim HP1 ,φ + 3 − ν φ(N ) , where for a finite subgroup G ⊆ PGL(2, K ), ν(G) = dim NorPGL(2,K ) (G)

(8.3.1)

464

CORNELISSEN and KATO

is the dimension of the normalizer of G in PGL(2, K ) as an algebraic group. Formula (8.3.1) continues to hold when hulls are replaced by tangent spaces. By Dickson’s Hauptsatz (see Theorem 8.3.3), the finite groups N acting on P1 in positive characteristic are known. Let us first set up the notation. Notation 8.3.2 We let Dn denote the dihedral group of order 2n. We write P(2, q) to denote either PGL(2, q) or PSL(2, q) by slight abuse of notation, with the convention that any related numerical quantities that appear between set delimiters {} are only to be considered for PSL(2, q). We now recall this classification in the version as it is given in R. Valentini and M. Madan [21], as this more geometrical form immediately allows us to compute D N ,φ using the results from Section 5. THEOREM 8.3.3 (Dickson’s Hauptsatz; see [21, Theorem 1]) Any finite subgroup of PGL(2, K ) is isomorphic to a finite subgroup of PGL(2, p m ) for some m > 0. The group PGL(2, p m ) has the following finite subgroups G, such that πG is branched over d points with ramification groups isomorphic to G 1, . . . , G d : (i) G = Z/nZ for (n; p) = 1, d = 2, and G 1 = G 2 = Z/nZ; (ii) G = Dn with p 6= 2, n| p m ± 1, d = 3, and G 1 = G 2 = Z/2Z, G 3 = Z/nZ or, also, p = 2, (n; 2) = 1, d = 2, and G 1 = Z/2Z, G 2 = Z/nZ; (iii) G = (Z/ pZ)t o Z/nZ for t ≤ m and n| p m − 1, n| p t − 1 with d = 2, and G 1 = G, G 2 = Z/nZ if n > 1, and d = 1, G 1 = G otherwise; (iv) G = P(2, p t ) with d = 2, and G 1 = (Z/ pZ)t o Z/{1/2}( p t − 1)Z, G 2 = Z/{1/2}( p t + 1)Z; (v) G = A4 if p 6= 2, 3, d = 3, and G 1 = Z/2Z, G 2 = G 3 = Z/3Z; (vi) G = S4 if p 6= 2, 3, d = 3, and G 1 = Z/2Z, G 2 = G 3 = Z/4Z; (vii) G = A5 if 5| p 2m − 1, p 6= 2, 3, 5 with d = 3, and G 1 = Z/2Z, G 2 = Z/3Z, G 3 = Z/5Z, or p = 3, d = 2, and G 1 = Z/3Z o Z/2Z, G 2 = Z/5Z.

8.3.4 The normalizer of a finite subgroup N of PGL(2, K ) has dimension ν(N ) = 0 unless N is cyclic of order prime to p, in which case ν(N ) = 1 or, if N is a pure p-group, ν(N ) = 2. LEMMA

Proof Any group from the above list which does not belong to the mentioned exceptions has

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

465

Table 2

G Z/nZ Dn (Z/ pZ)t

(Z/ pZ)t o Z/nZ P(2, p t ) A4 , S4 A5

( p, t, n) (n; p) = 1 p 6= 2 p=2 p 6= 2, 3 p=3 p = 2, t > 1 p = 2, t = 1 p 6= 2 and n 6= 2 or p = 2, 3 n=2 { p t 6= 5} p= 6 3 p=3

h(G) 2 3 4 t t t −1 2 t/s + 2 t +2 3 3 3 3

t (G) 2 3 4 t +1 t t 2 t/s + 2 t +3 3 3 3 4

at least three fixed points on P1 , the set of which should also remain stable under the action of the normalizer of N , which hence is finite. A cyclic subgroup N of order prime to p has a diagonalizable generator, and by a direct computation, this is seen to be exactly stabilized by the one-dimensional group
D generated by the center of PGL(2, K ) and the involution 01 10 . A p-group N can be put into upper diagonal form by conjugation, and a little computation shows that the stabilizer of such a group consists precisely of the twodimensional group of upper trigonal matrices. 8.4 Let N be as in Section 7, and suppose a Bass-Serre representation of N is given as in Theorem 7.5. Then X X dim H N∼,φ = 3c(TN ) − 3 + h(Nv ) − h(Ne ) THEOREM

v∈V (TN )

e∈E(TN )

and
dim K D ∼ N ,φ K [] = 3c(TN ) − 3 +

X v∈V (TN )

t (Nv ) −

X

t (Ne ),

e∈E(TN )

where for a finite group G ⊂ PGL(2, K ), the numbers h(G) and t (G) are given in Table 2.

466

CORNELISSEN and KATO

Table 3 G Z/nZ Dn (Z/ pZ)t

(Z/ pZ)t o Z/nZ P(2, p t ) A4 , S4 A5

( p, t, n) (n; p) = 1 p 6= 2 p=2 p 6= 2, 3 p=3 p = 2, t > 1 p = 2, t = 1 p 6= 2 and n 6= 2 or p = 2, 3 n=2 { p t 6= 5}∗ p 6= 3 p=3

h alg (G) 0 0 1 t −1 t −1 t −2 1 t/s − 1 t −1 0 0 0 0

t alg (G) 0 0 1 t t −1 t −1 1 t/s − 1 t 0 0 0 1

3 − ν(G) 2 3 3 1 1 1 1 3 3 3 3 3 3

Proof Since N is infinite, the action of PGL(2)∧ is of dimension 3. By Proposition 8.1 and Section 8.2, we are reduced to computing D N ,φ for finite N occuring in TN . We know which different G can occur on the edges and vertices of TN by Dickson’s theorem (Theorem 8.3.3). For each of these, using (8.3.1), we are reduced to the computation of the algebraic data, for which we appeal to Theorem 5.1, and to the computation of ν(G), which is in Lemma 8.3.4. If we let h alg (G) and t alg (G) denote the Krull dimension of the prorepresentable hull and vector-space dimension of the tangent space to DP1 ,φ|G , respectively, we have Table 3. In this computation, note at ∗ that PSL(2, 5) = A5 does not occur in Dickson’s list if p = 5. Remark 8.4.1 The main analytic theorem stated in the introduction follows from Theorem 8.4 by excluding the cases g ≤ 2, p = 2, 3.

9. Compatibility between algebraic and analytic deformation 9.1. Deformation of Mumford uniformization We have already seen how to compare analytic and algebraic deformation functors for finite groups acting on P1 . We now want to compare these functors in general, in particular to achieve equality between the apparently different results from the main

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

467

algebraic and analytic theorems (Theorems 5.1 and 8.4). Let X be a Mumford curve over K uniformized by a Schottky group 0, and let φ : N ,→ PGL(2, k) be a discrete group between 0 and its normalizer in PGL(2, K ). Let ρ : N / 0 ,→ Aut(X ). To be able to compare the functors D ∼ N ,φ and D X,ρ , it is necessary to develop a theory of algebraic deformation of Mumford uniformization; this just means to translate the formalism of Mumford [16] from fields K to elements in the category C K . For lack of a reference, we sketch it here; the reader who wants to follow the details is encouraged to take a copy of [16] to hand. 9.2. Analytic objects in C K The maximal ideal of an object A in C K is denoted by m A . Each object A in C K can be made into a K -affinoid algebra in a unique way by a suitable surjective homomorphism K hX 1 , . . . , X n i → A over K (see [2, (6.1)]). We denote by O A (resp., mO A ) the subring (resp., the ideal in O A ) consisting of power-bounded (resp., topologically nilpotent) elements in A (see [2, (6.2.3)]). Since every element in m A is nilpotent, we have O A ∩ m A ⊆ mO A . By this, it is easily seen that O A is a local ring with the maximal ideal mO A , that O A /mO A ∼ = k, and that π A−1 (O K ) = O A and π A−1 (mO K ) = mO A , where π A : A → K is the reduction map. Example In the ring of dual numbers K [], the ring of power-bounded elements is O K + K , whereas the ideal of topologically nilpotent elements is mO K + K . 9.3. Lattices Let A be an object in C K . By a lattice in A2 we mean an O A -submodule M in A2 that is free of rank 2. By elementary commutative algebra, this is equivalent to M ⊂ A2 being an O A -submodule such that the image M in K 2 by the reduction map A2 → (0) K 2 is a lattice in the usual sense. We consider the set 1 A of similarity classes of (0) lattices up to multiplication by A∗ . Then 1 A can be naturally identified with the set of equivalence classes of couples (P, φ), where P is an O A -scheme isomorphic to P1O A and φ is an isomorphism between P ⊗ A and P1A , and two couples (P, φ) and (P0 , φ 0 ) are equivalent if there exists an O A -isomorphism ψ : P → P0 such that φ 0 ◦ ψ = φ. (0) The identification between 1 A and the space of such couples is given by M 7→ P(M) = Proj(SymO A M), where φ is induced from M ⊗ A ∼ = A2 . 9.4. Trees We take a subgroup N ⊂ PGL(2, A) such that its image N in PGL(2, K ) by the reduction map is finitely generated, discrete, and isomorphic to N . Such a subgroup

468

CORNELISSEN and KATO

N contains a normal free subgroup 0 of finite index since N does and N and N are isomorphic as groups. This 0 satisfies a flatness condition analogous to [16, (1.4)] (or, equivalently, property ∗ in [16, p. 139]) in the following sense: If 6 is the set of all sections Spec A → P1A fixed by nontrivial elements γ ∈ 0, then for any P1 , P2 , P3 , P4 ∈ 6, the cross-ratio R := R(P1 , P2 ; P3 , P4 ) or its inverse R −1 lies in O A . (Note that 6 does not depend on the choice of 0 in N .) The proof follows easily from the fact that π A−1 (O K ) = O A . Given P1 , P2 , P3 with homogeneous coordinates w1 , w2 , w3 , respectively, let M = O A a1 w1 + O A a2 w2 + O A a3 w3 , where the ai satisfy a nontrivial linear relation a1 w1 + a2 w2 + a3 w3 = 0. The class v(P1 , P2 , P3 ) of M in (0) (0) 1 A depends only on P1 , P2 , P3 . We let 10 be the set of all such v(P1 , P2 , P3 ). The (0) set 10 is linked in the sense of [16, (1.11)], and the tree thus obtained is obviously the usual tree with respect to 0. 9.5 The construction of the formal scheme also parallels the original one. For M1 and M2 (0) in 10 , one defines the join P(M1 ) ∨ P(M2 ) to be the closure of the graph of the birational map P(M1 ) → · · · → P(M2 ) induced from φ2−1 ◦ φ1 , where (P(Mi ), φi ) corresponds to Mi (i = 1, 2) by the correspondence from Section 9.3. The formal scheme P0 over Spf O A is then constructed as in [16, p. 156] using these joins. Obviously, its fiber over Spf O K is isomorphic to the usual formal scheme; in particular, their underlying topological spaces are isomorphic. It is clear that the associated rigid space 0 of P0 in the sense of [3, Section 5] is the complement in P1,an A of the closure of the set of fixed rig-points (corresponding to the fixed sections). The quotient and the algebraization can equally well be taken by reasoning as in the usual case. What finally comes out is a scheme X 0 over A with special fiber over K the Mumford curve corresponding to 0, and hence one can further take a finite quotient by N / 0. 9.6 The above construction of infinitesimal deformation of Mumford uniformization induces a morphism of functors 8 : D∼ N ,φ −→ D X,ρ by associating to a deformation of N ,→ PGL(2, K ) to N¯ ,→ PGL(2, A) the corresponding “Mumford” curve over Spec A. By an argument parallel to that of [16, Section 4], it is not difficult to see the following. PROPOSITION 9.7 The morphism 8 is an isomorphism.

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

469

Remark 9.7.1 If X is a Mumford curve uniformized by a Schottky group 0 and N is between 0 and its normalizer in PGL(2, K ), let ρ : N \0 → Aut(X ) be the corresponding representation. Then the (a priori very different looking) results from the algebraic computation in Theorem 5.1 for D X,ρ and the analytic computation in Theorem 8.4 for D ∼ N ,φ agree. There may be a more direct combinatorial proof of this equality. Example 9.7.2 (Artin-Schreier-Mumford curves) If, for the Artin-Schreier curves of Example 5.1.2 over a nonarchimedean field K , the value of c satisfies |c| < 1, then X t,c is a Mumford curve, and the corresponding normalizer of its Schottky group is
Nt = (Z/ pZ)t o Z/( p t − 1)Z ∗Z/( pt −1)Z D pt −1 . We compute from this that the analytic infinitesimal deformation space is 1-dimensional, in concordance with the algebraic result. Example 9.7.3 (Drinfeld modular curves) The Drinfeld modular curve X (n) is known to be a Mumford curve (cf. [8]), and the normalizer of its Schottky group is isomorphic to an amalgam (cf. [5]) N (n) = PGL(2, p t ) ∗(Z/ pZ)t oZ/( pt −1)Z (Z/ pZ)td o Z/( p t − 1)Z (at least if p 6= 2, q 6= 3). The above formula gives a (d −1)-dimensional infinitesimal analytic deformation space, and this agrees with the algebraic result. Acknowledgments. G. Cornelissen is an honorary postdoctoral fellow of the Fund for Scientific Research–Flanders (FWO-Vlaanderen). This work was done while Cornelissen was visiting Kyoto University and the Max-Planck-Institut f¨ur Mathematik (MPIM) in Bonn and while F. Kato was visiting the University of Paris-VI. Thanks to Frans Oort for pointing us to [1]. References [1]

´ J. BERTIN and A. MEZARD , D´eformations formelles des revˆetements sauvagement

[2]

ramifi´es de courbes alg´ebriques, Invent. Math. 141 (2000), 195 – 238. MR 2001f:14023 433, 434, 437, 439, 440, 441, 448, 449, 450, 452, 453, 455, 457, 469 ¨ S. BOSCH, U. GUNTZER, and R. REMMERT, Non-Archimedean Analysis, Grundlehren Math. Wiss. 261, Springer, Berlin, 1984. MR 86b:32031 467 ¨ S. BOSCH and W. LUTKEBOHMERT , Formal and rigid geometry, I: Rigid spaces, Math. Ann. 295 (1993), 291 – 317. MR 94a:11090 468

[3]

470

CORNELISSEN and KATO

[4]

K. S. BROWN, Cohomology of Groups, Grad. Texts in Math. 87, Springer, New York,

[5]

G. CORNELISSEN, F. KATO, and A. KONTOGEORGIS, Discontinuous groups in positive

1982. MR 83k:20002 444, 447

[6] [7] [8] [9] [10] [11]

[12]

[13] [14] [15] [16] [17] [18] [19] [20] [21]

characteristic and automorphisms of Mumford curves, Math. Ann. 320 (2001), 55 – 85. CMP 1 835 062 432, 433, 435, 458, 459, 460, 469 W. DICKS and M. J. DUNWOODY, Groups Acting on Graphs, Cambridge Stud. Adv. Math. 17, Cambridge Univ. Press, Cambridge, 1989. MR 91b:20001 461 H. M. FARKAS and I. KRA, Riemann Surfaces, Grad. Texts in Math. 71, Springer, Berlin, 1980. MR 82c:30067 432 E.-U. GEKELER and M. REVERSAT, Jacobians of Drinfeld modular curves, J. Reine Angew. Math. 476 (1996), 27 – 93. MR 97f:11043 432, 469 L. GERRITZEN and M. VAN DER PUT, Schottky Groups and Mumford Curves, Lecture Notes in Math. 817, Springer, Berlin, 1980. MR 82j:10053 459, 460 A. GROTHENDIECK, Sur quelques points d’alg`ebre homologique, Tˆohoku Math. J. (2) 9 (1957), 119 – 221. MR 21:1328 433, 437, 438, 440 , “G´eom´etrie formelle et g´eom´etrie alg´ebrique” in S´eminaire Bourbaki, Vol. 5, exp. no. 182, Soc. Math. France, Montrouge, 1995, 193 – 220, errata p. 390. CMP 1 603 467 432 F. HERRLICH, “On the stratification of the moduli space of Mumford curves” in Groupe d’´etude d’analyse ultram´etrique: 11`eme ann´ee: 1983/84, Secr´etariat Math., Paris, 1985, exp. no. 18. MR 87m:14029 436 , Nichtarchimedische Teichm¨ullerr¨aume, Nederl. Akad. Wetensch. Indag. Math. 49 (1987), 145 – 169. MR 88m:32045 436 N. KOBLITZ, p-adic variation of the zeta-function over families of varieties defined over finite fields, Compositio Math. 31 (1975), 119 – 218. MR 54:2658 438 R. LIDL and H. NIEDERREITER, Finite Fields, 2d ed. Encyclopedia Math. Appl. 20, Cambridge Univ. Press, Cambridge, 1997. MR 97i:11115 439 D. MUMFORD, An analytic construction of degenerating curves over complete local rings, Compositio Math. 24 (1972), 129 – 174. MR 50:4592 458, 459, 467, 468 S. NAKAJIMA, p-ranks and automorphism groups of algebraic curves, Trans. Amer. Math. Soc. 303 (1987), 595 – 607. MR 88h:14037 438, 457 M. SCHLESSINGER, Functors of Artin rings, Trans. Amer. Math. Soc. 130 (1968), 208 – 222. MR 36:184 431, 440 J.-P. SERRE, Corps Locaux, 2d ed., Publ. Inst. Math. Univ. Nancago 8, Actualit´es Sci. Indust. 1296, Hermann, Paris, 1968. MR 50:7096 438, 445 , Trees, Springer, Berlin, 1980. MR 82c:20083 460, 461 R. C. VALENTINI and M. L. MADAN, A Hauptsatz of L. E. Dickson and Artin-Schreier extensions, J. Reine Angew. Math. 318 (1980), 156 – 177. MR 82e:12030 464

Cornelissen Department of Mathematics, Utrecht University, Budapestlaan 6, 3584 CD Utrecht, The Netherlands; [email protected]

EQUIVARIANT DEFORMATION IN POSITIVE CHARACTERISTIC

Kato Department of Mathematics, Faculty of Science, Kyoto University, Kyoto 606-8502, Japan; [email protected]

471

ZERO-ENTROPY ALGEBRAIC Zd -ACTIONS THAT DO NOT EXHIBIT RIGIDITY SIDDHARTHA BHATTACHARYA

Abstract We show that there exist mixing zero-entropy algebraic Z8 -actions that are measurably and topologically conjugate but not algebraically conjugate. This result gives the first known examples of mixing zero-entropy algebraic Zd -actions that do not have rigidity properties, and it provides a negative answer to the isomorphism rigidity problem raised in B. Kitchens and K. Schmidt (Isomorphism rigidity of irreducible algebraic Zd -actions, Invent. Math. 142 (2000), 559 – 577) and Schmidt (“The dynamics of algebraic Zd -actions” in European Congress of Mathematics (Barcelona, 2000), Vol. 1, Progr. Math. 201, Birkh¨auser, Basel, 2001, 543 – 553). 1. Introduction An algebraic Zd -action is an action α : n → α(n) of Zd on a compact abelian group X by continuous automorphisms of X . It is easy to see that any such action preserves λ X , the Haar measure on X . If α is a homomorphism from Zd to GL(n, Z) for some n ≥ 1, then the natural action of α(Zd ) on Rn induces an algebraic Zd -action on Tn ∼ = Rn /Zn . Another class of examples is given by group shifts. Let F be a finite d abelian group, and let S be the shift action of Zd on F Z defined by S(m)(x)(n) = x(n + m),

∀m, n ∈ Zd . d

A group shift is a closed shift-invariant subgroup X ⊂ F Z , together with the shift action of Zd restricted to X . If (X, α) and (Y, β) are two algebraic Zd -actions, then a measurable map f : X → Y is said to be a measurable conjugacy if f is a measure space isomorphism from (X, λ X ) to (Y, λY ) and for all n ∈ Zd , f ◦α(n) = β(n)◦ f a.e. λ X . A topological conjugacy (resp., algebraic conjugacy) from (X, α) to (Y, β) is a homeomorphism (resp., a continuous isomorphism) θ from X to Y which satisfies θ ◦ α(n) = β(n) ◦ θ for all n in Zd . If X and Y are compact abelian groups and f : X → Y is a measurable map, then f is said to be affine if there exists an element c ∈ Y and a continuous DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 3, Received 24 October 2001. Revision received 18 December 2001. 2000 Mathematics Subject Classification. Primary 37A35; Secondary 22D40, 28D15. Author’s work supported by a postdoctoral fellowship from the Hebrew University of Jerusalem. 471

472

SIDDHARTHA BHATTACHARYA

surjective group homomorphism θ : X → Y such that f (x) = c + θ (x) a.e. λ X . If (X, α) is an algebraic Zd -action, then α has completely positive entropy if and only if it is Bernoulli (cf. [6]), and α has zero entropy if and only if it does not admit Bernoulli factors (cf. [4]). In the former case, entropy is a complete metric invariant. On the other hand, zero-entropy actions tend to exhibit remarkable rigidity properties. For various classes of mixing zero-entropy algebraic Zd -actions, it has been established that measurable or topological conjugacy implies algebraic conjugacy (see [1], [3], [5], [7]). The following conjecture in this direction has been raised in [9, Conj. 3.5]. CONJECTURE

Let d > 1, and let α and β be mixing algebraic Zd -actions on compact abelian groups X and Y , respectively. If α has zero entropy and if φ : X → Y is a measurable conjugacy of α and β, then φ is λ X a.e. equal to an affine map. In particular, measurable conjugacy implies algebraic conjugacy. This conjecture is known to hold for expansive actions on tori (see [1]) and irreducible actions on zero-dimensional groups (see [3]). The results in [5] and [7] show that the topological counterpart of this conjecture holds for several classes of algebraic Z2 actions on zero-dimensional groups. For various related results and questions, the reader is referred to [2], [3], and [5]. In this paper we give a counterexample to the above conjecture in the general case. More specifically, we prove the following result. 1.1 There exist mixing zero-entropy algebraic Z8 -actions (X 1 , α1 ) and (X 2 , α2 ) such that the actions α1 and α2 are measurably and topologically conjugate but not algebraically conjugate. THEOREM

We note that this result can be used to construct examples of mixing zero-entropy algebraic Zd -actions that do not have other rigidity properties. For example, if α1 and α2 are as above, then a standard skew-product construction shows that the action α1 × α2 admits nonaffine elements in the measurable and topological centralizer. Using an argument used in [3], one can also show that the action α1 × α2 admits nontrivial joinings. 2. Nonrigid actions d For any d ≥ 1, we denote by X d the group (Z/2Z)Z , equipped with pointwise addition and the topology of pointwise convergence. It is easy to see that X d is a compact

NONRIGID ALGEBRAIC Zd -ACTIONS

473

zero-dimensional abelian group. A Markov subgroup of X d is a closed subgroup that is invariant under the shift action. In [2] it was shown that the dynamics of the shift action on Markov subgroups can be studied using algebraic methods. We briefly recall the results that are needed in our construction. Let F2 denote the field with two elements, and for d ≥ 1, let Rd denote the group-ring of Zd with coefficients in F2 . The ring Rd can be identified with the ring of Laurent polynomials in d commuting variables with coefficients in F2 . Every element p ∈ Rd is written as X p= c p (m)u m m∈Zd

with u m = u m 1 · · · u m d and c p (m) ∈ F2 , where c p (m) = 0 for all but finitely many m. The shift action S induces an Rd -module structure on X d defined by p · x = P c p (m)S(m)(x). For any Markov subgroup H of X d , we define I (H ) ⊂ Rd by  I (H ) = p ∈ Rd p · x = 0, ∀x ∈ H . It is easy to see that I (H ) is an ideal in Rd . Using duality theory of compact abelian groups, it can be shown that the correspondence H 7→ I (H ) is an order-reversing bijection from the set of all Markov subgroups of X d to the set of all ideals in Rd . We recall two basic facts about this correspondence. For proofs, the reader is referred to [8, Th. 6.5, Prop. 19.4]. PROPOSITION 2.1 Let d ≥ 1, and let H ⊂ X d be a Markov subgroup. (1) The action (H, S) has zero entropy if and only if I (H ) is nonzero (i.e., H is a proper subgroup). (2) If I (H ) is a prime ideal, then the action (H, S) is mixing if and only if for every nonzero m, u m − 1 does not lie in I (H ).

For any x, y ∈ X d , we define their product x ? y ∈ X d by x ? y(i) = x(i) y(i). It is easy to see that X d becomes a compact topological ring with respect to this product, and S(i) is a ring automorphism for all i in Zd . Let H, K ⊂ X d be Markov subgroups such that for all x, y in H , x ? y ∈ K . We consider two different group structures on the compact set H × K . The first one is the usual componentwise addition, and the other one is defined by (h 1 , k1 ) (h 2 , k2 ) = (h 1 + h 2 , h 1 ? h 2 + k1 + k2 ). It is easy to verify that H × K becomes a compact zero-dimensional group with respect to both group structures. Furthermore, since each S(i) is a ring automorphism of X d , it follows that the Zd -action S × S on H × K is an action by automorphisms

474

SIDDHARTHA BHATTACHARYA

with respect to both group structures. We denote the resulting algebraic Zd -actions by (X 1 , α1 ) and (X 2 , α2 ), respectively. PROPOSITION 2.2 Let H, K ⊂ X d be proper Markov subgroups such that the actions (H, S) and (K , S) are mixing, and for all x, y in H , x ? y ∈ K . Then α1 and α2 are mixing zeroentropy algebraic Zd -actions that are measurably and topologically conjugate but not algebraically conjugate.

Proof We note that every nonzero element of X 1 has order 2. On the other hand, for any nonzero element x in H , the element (x, 0) lies in X 2 and (x, 0) (x, 0) = (0, x ? x) = (0, x) 6= 0. This shows that X 1 and X 2 are not isomorphic as abstract groups, and in particular, the actions α1 and α2 are not algebraically conjugate. If i : X 1 → X 2 denotes the map induced by the identity map on H × K , then it is easy to see that i is a topological conjugacy from (X 1 , α1 ) to (X 2 , α2 ). For i = 1, 2, let λi denote the Haar measure on X i , viewed as a measure on H × K . We note that for any x0 in H × K , the map x 7→ x0 x induces an invertible affine transformation on the group X 1 . Hence the measure λ1 is preserved by the maps that are translations with respect to the group structure given by ; that is, λ1 = λ2 . This proves that the map i is also a measurable conjugacy. Since H and K are proper subgroups and since shift actions on H and K are mixing, it follows that the action α1 is mixing and has zero entropy. Since α1 and α2 are measurably conjugate, this completes the proof. For any subspace C ⊂ Fd2 , we define a Markov subgroup X C ⊂ X d by  X C = x ∈ X d (x(i + e1 ), . . . , x(i + ed )) ∈ C, ∀i ∈ Zd , where e1 , . . . , ed are the standard unit vectors in Zd . The associated ideal I (X C ) can be described as follows. For any v in Fd2 , we define a polynomial pv in Rd as P pv = dj=1 v j u j . If v and w are two elements of Fd2 , then their dot product v · w P is defined by v · w = vi wi . For any set A ⊂ Fd2 , we denote by A⊥ the subspace consisting of all vectors v satisfying v · w = 0 for all w in A. We note that pv · x = 0 for any v ∈ C ⊥ and x in X C . Since (C ⊥ )⊥ = C, I (X C ) is the ideal generated by the set { pv | v ∈ C ⊥ }. For any fixed d ≥ 1, we define a map B : Fd2 × Zd → Z by X B(n, v) = ni . i:vi =1

NONRIGID ALGEBRAIC Zd -ACTIONS

475

We denote the vector (1, . . . , 1) ∈ Fd2 by 1. 2.3 Let C ⊂ Fd2 be a subspace such that 1 ∈ C, and for every nonzero n ∈ Zd , there exists a v ∈ C satisfying B(n, v) 6= 0. Then the Zd -action (X C , S) is mixing. LEMMA

Proof We claim that I (X C ) is a prime ideal of Rd . Let Pd be the polynomial ring in dvariables with coefficients in F2 . We can identify Pd with the subring of Rd which consists of all p such that c p (n) = 0 whenever n i < 0 for some i. For any set B ⊂ Fd2 , let J B ⊂ Pd denote the ideal generated by the set { pv | v ∈ B}. Let C1 ⊂ Fd2 denote the subspace consisting of all v = (v1 , . . . , vd ) such that vi = 0 whenever i > dim(C). We choose a linear automorphism of θ of Fd2 such that θ (C1 ) = C. Then θ (JC1 ) = JC , where θ is the automorphism of Pd induced by θ . Since Pd /JC1 is isomorphic to Pd−k , it follows that JC1 and JC are prime ideals of Pd . Let p1 , p2 be P two elements of Rd such that p1 p2 ∈ I (X C ). Then p1 p2 can be written as A pv qv , where each qv is an element of Rd . We choose n ∈ Zd such that u n p1 , u n p2 , and each u 2n qv lie in Pd . Then u 2n p1 p2 ∈ JC , which implies that either u n p1 ∈ JC or u n p2 ∈ JC . Since JC ⊂ I (X C ) and each u n is invertible in Rd , this proves the claim. Now for any v ∈ Fd2 , let φv be the unique homomorphism from Rd to R1 = F2 [z, z −1 ] such that φv (u i ) = z if vi = 1 and φv (u i ) = 1 if vi = 0. Let v be any element of C ⊥ , and let w be any element of C. Since C contains 1 and v · w = 0, the sets {i | vi = wi = 1} and {i | vi = 1, wi = 0} contain an even number of elements. Therefore X φw ( p v ) = vi φw (u i ) = 0. This shows that I (X C ) ⊂ ker(φw ) for all w in C. Let n be any nonzero element of Zd . By our assumption, there exists a w in C such that B(n, w) is nonzero. Since φw (u n ) = z B(n,w) , we conclude that u n − 1 does not lie in I (X C ). Now from the above claim and Proposition 2.1, it follows that the action (X C , S) is mixing. Proof of Theorem 1.1 For any v in F82 , let |v| denote the number of nonzero coordinates of v. Let E ⊂ F82 denote the subspace consisting of all v such that |v| is even, and let C ⊂ F82 denote the subspace generated by B = {v1 , . . . , v4 }, where v1 , . . . , v4 are given by v1 = (11110000),

v2 = (00111100),

v3 = (00001111),

v4 = (10101010).

Since the dot product is bilinear and B ⊂ B ⊥ , it follows that v·w = 0 for all v, w in C. We note that if v, w are two elements of F82 such that v·w = 0 and |v|, |w| are divisible

476

SIDDHARTHA BHATTACHARYA

by 4, then |v + w| is also divisible by 4. Thus, every element of C has zero, four, or eight nonzero coordinates. We claim that for any nonzero n ∈ Z8 , there exists v in C such that B(n, v) 6= 0. We choose 1 ≤ i, j ≤ 8 such that n i + n j 6= 0. Let φ be the vector space homomorphism from C to F22 defined by φ(v) = (vi , v j ), and let A ⊂ C be the set consisting of all v such that φ(v) = (1, 1). Since C has 16 elements and F22 has four elements, the set A = 1 + ker(φ) contains at least four elements. We choose two distinct vectors w, w0 in A such that |w| = |w0 | = 4. As w · w0 = 0, we see that {k | wk = w0k = 1} = {i, j}. Hence 2(n i + n j ) = B(n, w) + B(n, w0 ) − B(n, w + w0 ), and since n i + n j 6= 0, this proves the claim. Since 1 ∈ C and C ⊂ E, from the above claim and Lemma 2.3 it follows that the Z8 -actions (X C , S) and (X E , S) are mixing. As v · w = 0 for all v, w ∈ C, it is easy to see that for any x, y ∈ X C , x ? y ∈ X E . Now Theorem 1.1 follows from Proposition 2.2. References [1] [2]

[3] [4]

[5] [6]

[7]

[8] [9]

A. KATOK, S. KATOK, and K. SCHMIDT, Rigidity of measurable structure for Zd -actions by automorphisms of a torus, preprint, arXiv:math.DS/0003032 472 2 B. KITCHENS and K. SCHMIDT, “Markov subgroups of (Z/2Z)Z ” in Symbolic

Dynamics and Its Applications (New Haven, Conn., 1991), Contemp. Math. 135, Amer. Math. Soc., Providence, 1992, 265 – 283. MR 93k:58136 472, 473 , Isomorphism rigidity of irreducible algebraic Zd -actions, Invent. Math. 142 (2000), 559 – 577. MR 2001j:37004 472 D. LIND, K. SCHMIDT, and T. WARD, Mahler measure and entropy for commuting automorphisms of compact groups, Invent. Math. 101 (1990), 593 – 629. MR 92j:22013 472 A. N. QUAS and P. B. TROW, Mappings of group shifts, Israel J. Math. 124 (2001), 333 – 365. MR 2002e:37020 472 D. J. RUDOLPH and K. SCHMIDT, Almost block independence and Bernoullicity of Zd -actions by automorphisms of compact abelian groups, Invent. Math. 120 (1995), 455 – 488. MR 96d:22004 472 M. A. SHERESHEVSKY, On the classification of some two-dimensional Markov shifts with group structure, Ergodic Theory Dynam. Systems 12 (1992), 823 – 833. MR 93k:58082 472 K. SCHMIDT, Dynamical Systems of Algebraic Origin, Progr. Math. 128, Birkh¨auser, Basel, 1995. MR 97c:28041 473 , “The dynamics of algebraic Zd -actions” in European Congress of Mathematics (Barcelona, 2000), Vol. 1, Progr. Math. 201, Birkh¨auser, Basel, 2001, 543 – 553. CMP 1 905 342 472

School of Mathematics, Tata Institute of Fundamental Research, Colaba, Mumbai 400005, India; [email protected]

TENSOR PRODUCT VARIETIES AND CRYSTALS: THE ADE CASE ANTON MALKIN

Abstract Let g be a simple simply laced Lie algebra. In this paper two families of varieties associated to the Dynkin graph of g are described: tensor product and multiplicity varieties. These varieties are closely related to Nakajima’s quiver varieties and should play an important role in the geometric constructions of tensor products and intertwining operators. In particular, it is shown that the set of irreducible components of a tensor product variety can be equipped with the structure of a g-crystal isomorphic to the crystal of the canonical basis of the tensor product of several simple finitedimensional representations of g, and that the number of irreducible components of a multiplicity variety is equal to the multiplicity of a certain representation in the tensor product of several others. Moreover, the decomposition of a tensor product into a direct sum is described geometrically. Contents 0. Introduction . . . . . . . . . . . . . . . . . . . . . . 1. Crystals . . . . . . . . . . . . . . . . . . . . . . . . 1.1. Weights and roots . . . . . . . . . . . . . . . . 1.2. Definition of g-crystals . . . . . . . . . . . . . . 1.3. Tensor product of g-crystals . . . . . . . . . . . 1.4. Highest-weight crystals and closed families . . . 1.5. Uniqueness theorem . . . . . . . . . . . . . . . 2. Nakajima varieties and tensor product varieties . . . . . 2.1. Oriented graphs and path algebras . . . . . . . . 2.2. Quiver varieties . . . . . . . . . . . . . . . . . 2.3. Stability and G V -action . . . . . . . . . . . . . 2.4. The variety Ms (d, v0 , v) . . . . . . . . . . . . . 2.5. The set M (d, v0 , v) . . . . . . . . . . . . . . . 2.6. Tensor product varieties and multiplicity varieties DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 3, Received 29 March 2001. Revision received 24 January 2002. 2000 Mathematics Subject Classification. Primary 20G99.

477

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

478 483 483 483 484 485 486 486 486 488 489 490 491 492

478

ANTON MALKIN

2.7. Inductive construction of tensor product varieties . . . . . . . . . . 2.8. Dimensions of tensor product and multiplicity varieties . . . . . . . 2.9. Irreducible components of tensor product varieties . . . . . . . . . 2.10. The first bijection for a tensor product variety . . . . . . . . . . . 2.11. The multiplicity variety for two multiples . . . . . . . . . . . . . 2.12. The first bijection for a multiplicity variety . . . . . . . . . . . . . 2.13. The second bijection . . . . . . . . . . . . . . . . . . . . . . . . 2.14. The tensor decomposition bijection . . . . . . . . . . . . . . . . . 3. Levi restriction and the crystal structure on quiver varieties . . . . . . . . 3.1. A subquiver Q 0 . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2. The set Q Q 0 M (d, v0 , v) . . . . . . . . . . . . . . . . . . . . . . 3.3. Levi restriction . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4. Levi restriction and the second bijection for tensor product varieties 3.5. Levi restriction and the first bijection for tensor product varieties . . 3.6. Levi restriction and the tensor product decomposition . . . . . . . 3.7. Digression: The sl2 -case . . . . . . . . . . . . . . . . . . . . . . 3.8. A crystal structure on Q M (d, v0 ) . . . . . . . . . . . . . . . . . 3.9. The main theorem . . . . . . . . . . . . . . . . . . . . . . . . . 3.10. Corollary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.11. The extended Lie algebra g0 . . . . . . . . . . . . . . . . . . . . A. Appendix. Another description of multiplicity varieties and the tensor product diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.1. Lusztig’s description of the variety Ms,∗s D,V . . . . . . . . . . . . . A.2. Multiplicity varieties . . . . . . . . . . . . . . . . . . . . . . . . A.3. The tensor product diagram . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

494 495 497 498 500 501 503 504 506 506 506 507 508 509 511 512 515 517 517 518 520 520 521 522 522

0. Introduction 0.1 The purpose of this paper is to define and study two (closely related) families of quasiprojective varieties associated to a simply laced Dynkin graph D: the tensor product varieties and the multiplicity varieties. 0.2 In this paper, for the sake of clarity, the Dynkin graph D is assumed to be of ADE (finite simply laced) type. This condition can be relaxed by slightly adjusting the definitions of the varieties involved.

TENSOR PRODUCT VARIETIES AND CRYSTALS

479

Let g be the simple Lie algebra associated to D, and let g0 = g ⊕ t, where t is a Cartan subalgebra of g. Thus g0 is a reductive Lie algebra. The precise definition of the tensor product and multiplicity varieties is rather involved (see Sec. 2.6). For the purpose of this introduction, it is enough to know that a multiplicity variety n S(µ0 ; µ1 , . . . , µn ) is associated to a set µ0 , µ1 , . . . , µn , where µk belongs to a certain subset of the weight lattice of the reductive Lie algebra g0 (the set of integrable positive weights). A tensor product variety n T(µ1 , . . . , µn ; ν) is associated to a set µ1 , . . . , µn , ν, where µ1 , . . . , µn are as above, and ν is a weight of g0 . This way of writing parameters of n S and n T is for the purposes of the introduction only. Notation in the main body of the paper is different. In the case of n = 1, the variety 1 S(µ0 ; µ1 ) is empty unless µ0 = µ1 , in which reg case it coincides with Nakajima’s quiver variety M0 (v, w) (cf. [N1], [N2]). Here v and w can be expressed in terms of µ0 . Similarly, 1 T(µ1 ; ν) coincides with a fiber M(µ1 , ν) = M(v, v0 , w) of Nakajima’s resolution of singularities of the singular quiver variety (where again v, v0 , and w can be expressed in terms of µ1 and ν). 0.3 Multiplicity varieties and tensor product varieties have pure dimensions. Let n T (µ1 , . . . , µn ; ν) (resp., n S (µ0 ; µ1 , . . . , µn ), M (µ, ν)) be the set of irreducible components of the tensor product variety n T(µ1 , . . . , µn ; ν) (resp., the multiplicity variety n S(µ0 ; µ1 , . . . , µn ), Nakajima’s variety M(µ, ν)), and let F n F n T (µ1 , . . . , µn ) = 1 n ν T (µ , . . . , µ ; ν) , M (µ) = ν M (µ, ν). H. Nakajima [N2] (based on an idea of G. Lusztig [L1, Sec. 12]) has introduced the structure of a g0 -crystal on the set M (µ), and it has been shown by Y. Saito (based on his joint work with M. Kashiwara [KS]) that this crystal is isomorphic to the crystal of the canonical basis of the irreducible representation of g0 with highest weight µ. Strictly speaking, the above-mentioned authors consider g-crystals, but the extension to g0 is straightforward, and the weight lattice of g0 appears more naturally in geometry of quiver varieties. The main result of this paper is the construction of two bijections between sets of irreducible components (cf. Secs. 2.10, 2.13): αn : βn :

∼

T (µ1 , . . . , µn ) − → M (µ1 ) × · · · × M (µn ), ∼ Gn n T (µ1 , . . . , µn ) − → S (µ0 ; µ1 , . . . , µn ) × M (µ0 ).

n

µ0

Moreover, one has the following theorem (cf. Ths. 3.9, 3.11).

480

ANTON MALKIN

THEOREM

The composite bijection τn = βn ◦ αn−1 :

∼

M (µ1 ) ⊗ · · · ⊗ M (µn ) − →

M

n

S (µ0 ; µ1 , . . . , µn ) ⊗ M (µ0 )

µ0

is an isomorphism of g0 -crystals if one endows the sets M (µ1 ), . . . , M (µn ) with Nakajima’s crystal structure and considers the set n S (µ0 ; µ1 , . . . , µn ) as a trivial crystal. In Theorem 0.3, ⊗ (resp., ⊕) denotes the tensor product (resp., direct sum) of crystals, which coincides with the direct product (resp., disjoint union) on the level of sets. It follows, in particular, that the set n T (µ1 , . . . , µn ) can be equipped with the structure of a g0 -crystal using either the bijection αn or βn , and this crystal is isomorphic to the crystal of the canonical basis in the tensor product of n irreducible representations of g0 with highest weights µ1 , . . . , µn . The proof of Theorem 0.3 uses double reduction. First, it is shown that the statement for any n follows from the corresponding statement for n = 2. Then one uses restriction to Levi factors of parabolic subalgebras of g to reduce the problem to the sl2 -case. When n = 2 and g = sl2 , Theorem 0.3 becomes an elementary linear algebraic statement. 0.4 The existence of the crystal isomorphism τn allows one to apply a result of A. Joseph [J, Prop. 6.4.21] about the uniqueness of the family of crystals closed with respect to tensor products to give another proof of isomorphism between M (µ) and the crystal of the canonical basis of the irreducible representation of g0 with highest weight µ, and, more importantly, to prove that the set n S (µ0 ; µ1 , . . . , µn ) of irreducible components of a multiplicity variety has the cardinal equal to the multiplicity of the irreducible representation of g0 with highest weight µ0 in a direct sum decomposition of the tensor product of n irreducible representations with highest weights µ1 , . . . , µn . It is also equal to the corresponding multiplicity for the tensor product of representations of the simple Lie algebra g if one restricts the weights to the Cartan subalgebra of g. In other words, one has the following corollary of Theorem 0.3. COROLLARY

Let L(µ) denote the irreducible representation of g0 with highest weight µ. Then
|n S (µ0 ; µ1 , . . . , µn )| = dimC Homg0 L(µ0 ), L(µ1 ) ⊗ · · · ⊗ L(µn ) .

TENSOR PRODUCT VARIETIES AND CRYSTALS

481

0.5 Nakajima [N2] has constructed an action of g0 in the top (Borel-Moore) homology of varieties M(µ, ν). A direct generalization of his construction provides an action of g0 in the top homology of the tensor product varieties. Then the irreducible components of these varieties produce a distinguished basis in the representation space, which can be called, by analogy with [L5], a semicanonical basis in the tensor product. 0.6 The paper is organized as follows. Section 1 contains a review of Kashiwara’s theory of crystals. Section 2 begins with a description of Nakajima’s quiver varieties. Then the tensor product and multiplicity varieties are defined in Section 2.6. The remainder of Section 2 is devoted to various properties of these varieties. In particular, the tensor decomposition bijection is described in Section 2.14. In Section 3 it is shown (following Lusztig and Nakajima) how one can use restrictions to Levi factors of parabolic subalgebras of g to define crystal structures on the sets of irreducible components of the varieties involved. Then a variant of Theorem 0.3 (Th. 3.9) is proven using the Levi restriction. 0.7 Definitions of various varieties given in the main body of the paper are rather messy due to an abundance of indices. Unfortunately, one needs this notation in order to follow the arguments used in the proofs. However, in the appendix a more conceptual definition of the multiplicity varieties is given. Namely, they appear to be closely related to the Hall-Ringel algebra (cf. [R]) associated to a certain algebra F˜ introduced by Lusztig [L3], [L6]. The appendix also contains a diagram of varieties, called the tensor product diagram, which can be used to equip a certain category of perverse sheaves on a variety Z D described by Lusztig [L3] with a structure of a Tannakian category. 0.8 Special cases of the tensor product and multiplicity varieties have appeared in the literature. The most important case is related to a Dynkin graph of type A. It is known (see [KP], [N1]) that some quiver varieties associated to this Dynkin graph are related to partial resolution of singularities of the nilpotent cone in glk . Similarly, the multiplicity varieties in this case are related to Spaltenstein varieties for glk . (A Spaltenstein variety is a variety consisting of all parabolic subgroups P of a given type that contain a given nilpotent operator t ∈ glk and such that the projection of t to the Levi factor L of P belongs to a given nilpotent orbit in L.) A theorem due to P. Hall (cf. [H], [M, Chap. II]) implies that the number of irreducible components of

482

ANTON MALKIN

a Spaltenstein variety is equal to a certain multiplicity in the tensor product decomposition for gl N (k has no relation to N ), which is a special case of Corollary 0.4. To the best of the author’s knowledge, the tensor product varieties are new, even in the gl N -case. All known proofs of the Hall theorem are either combinatorial (through the Littlewood-Richardson rule) or use the relation between the tensor product for GL and the restriction for symmetric groups combined with results of W. Borho and R. MacPherson [BM]. The existence of the tensor product varieties, together with bijections αn and βn , provides a direct proof of the Hall theorem by showing the role of Spaltenstein varieties in the geometric theory of the tensor product. The gl N -case is described in [Ma2] without mentioning quiver terminology (purely in the language of flags and nilpotent orbits). In the case of the tensor product of n fundamental representations of gl N , the tensor product variety is a certain Lagrangian subvariety in the cotangent bundle of the variety studied by I. Grojnowski and Lusztig in [GL]. 0.9 Though the ground field is C throughout the paper, the multiplicity varieties are defined over arbitrary fields, and it is shown in [Ma1] that the number of Fq -rational points of n S (µ0 ; µ1 , . . . , µn ) is given by a polynomial in q with a leading coefficient equal to dimC Homg0 (L(µ0 ), L(µ1 ) ⊗ · · · ⊗ L(µn )) (cf. Cor. 0.4). This statement is a direct generalization of the Hall theorem (cf. [H], [M, Chap. II]) giving the number of Fq -rational points in a Spaltenstein variety for gl N . 0.10 A. Braverman and D. Gaitsgory [BG] have constructed crystals (together with the crystal tensor product) using certain “perverse cells” in the affine Grassmannian introduced by I. Mirkovi´c and K. Vilonen [MV]. The relation between quiver varieties and the affine Grassmannian in the non-type-A situation remains a mystery to the author. 0.11 Results similar to the ones described in this paper have been independently obtained by Nakajima [N4] and by M. Varagnolo and E. Vasserot [VV]. Nakajima has proven a statement analogous to Theorem 0.3, while Varagnolo and Vasserot have used a diagram similar to (A.3.1) to introduce a tensor product on a category closely related to perverse sheaves on the variety Z D (cf. Sec. 0.7). Lusztig [L6] has described a locally closed subset of a variety Z (no relation with Z D ) constructed by Nakajima (cf. [N2]), such that irreducible components of this subset (conjecturally) form a crystal isomorphic to the tensor product of two

TENSOR PRODUCT VARIETIES AND CRYSTALS

483

irreducible representations of g. The relation of Lusztig’s construction to the tensor product variety (for n = 2) is not discussed in this paper. 0.12 Throughout the paper the following conventions are used: the ground field is C; closed, locally closed, and so on, refer to the Zariski topology; fibration means locally trivial fibration, where locally refers to the Zariski topology; however, trivialization is analytic (not regular). 1. Crystals Crystals were unearthed by Kashiwara [K1], [K2], [K3]. An excellent survey of crystals, as well as some new results, was given by Joseph in [J, Chaps. 5, 6]. 1.1. Weights and roots Let g be a reductive or a Kac-Moody Lie algebra, and let I be the set of vertices of the Dynkin graph of g. It is assumed throughout the paper that g is simply laced. The weight lattice is denoted by Qg and is identified with Z[I ]. Let h, i be the natural inner product on Z[I ] given by X hv, ui = vi u i , i∈I

where vi denotes the ith component of v ∈ Z[I ]. ˆ corresponding to a simple Let A be the Cartan matrix of g. Then the simple root i, P weight i ∈ I , is given by Ai = j∈I A ji j ∈ Z≥0 [I ]. 1.2. Definition of g-crystals A g-crystal is a tuple (A , wt, {εi }i∈I , {ϕi }i∈I , {e˜i }i∈I , { f˜i }i∈I ), where • A is a set, • wt is a map from A to Qg , • εi and ϕi are maps from A to Z, • e˜i and f˜i are maps from A to A ∪ {0}. These data should satisfy the following axioms: ˆ ϕi (e˜i a) = ϕi (a) + 1, εi (e˜i a) = εi (a) − 1, for any • wt(e˜i a) = wt(a) + i, i ∈ I and a ∈ A such that e˜i a 6= 0; ˆ ϕi ( f˜i a) = ϕi (a) − 1, εi ( f˜i a) = εi (a) + 1, for any • wt( f˜i a) = wt(a) − i, i ∈ I and a ∈ A such that f˜i a 6= 0; • (wt(a))i = ϕi (a) − εi (a) for any i ∈ I and a ∈ A ; • f˜i a = b if and only if e˜i b = a, where i ∈ I , and a, b ∈ A . The maps e˜i and f˜i are called Kashiwara operators, and the map wt is called the weight function.

484

ANTON MALKIN

Remark. In a more general definition of crystals, the maps εi and ϕi are allowed to have infinite values. A g-crystal (A , wt, {εi }i∈I , {ϕi }i∈I , {e˜i }i∈I , { f˜i }i∈I ) is called trivial if wt(a) = 0

for any a ∈ A ,

εi (a) = ϕi (a) = 0

for any i ∈ I and a ∈ A ,

e˜i a = f˜i a = 0

for any i ∈ I and a ∈ A .

Any set A can be equipped with the trivial crystal structure as above. A g-crystal (A , wt, {εi }i∈I , {ϕi }i∈I , {e˜i }i∈I , { f˜i }i∈I ) is called normal if εi (a) = max{n | e˜in a 6= 0}, ϕi (a) = max{n | f˜in a 6= 0}, for any i ∈ I and a ∈ A . Thus in a normal g-crystal the maps εi and ϕi are uniquely determined by the action of e˜i and f˜i . A trivial crystal is normal. In the rest of the paper, all g-crystals are assumed to be normal, and thus the maps εi and ϕi are usually omitted. By abuse of notation, a g-crystal (A , wt, {εi }i∈I , {ϕi }i∈I , {e˜i }i∈I , { f˜i }i∈I ) is sometimes denoted simply by A . An isomorphism of two g-crystals A and B is a bijection between the sets A and B commuting with the action of the operators e˜i and f˜i , and the functions wt, εi , and ϕi . The direct sum A ⊕ B of two g-crystals A and B is their disjoint union as sets with the maps e˜i , f˜i , wt, εi , and ϕi acting on each component of the union separately. 1.3. Tensor product of g-crystals The tensor product A ⊗ B of two g-crystals A and B is their direct product as sets equipped with the following crystal structure: wt((a, b)) = wt(a) + wt(b), εi ((a, b)) = max{εi (a), εi (a) + εi (b) − ϕi (a)}, ϕi ((a, b)) = max{ϕi (b), ϕi (a) + ϕi (b) − εi (b)}, ( (e˜i a, b) if ϕi (a) ≥ εi (b), e˜i ((a, b)) = (a, e˜i b) if ϕi (a) < εi (b), ( ( f˜i a, b) if ϕi (a) > εi (b), f˜i ((a, b)) = (a, f˜i b) if ϕi (a) ≤ εi (b).

(1.3.1)

TENSOR PRODUCT VARIETIES AND CRYSTALS

485

Here (a, 0), (0, b), and (0, 0) are identified with zero. One can check that the set A × B with structure (1.3.1) satisfies all the axioms of a (normal) crystal and that the tensor product of crystals is associative. Remark. The tensor product of crystals is not commutative. Remark. The tensor product (in any order) of a crystal A with a trivial crystal B is L isomorphic to the direct sum b∈B Ab , where each Ab is isomorphic to A . 1.4. Highest-weight crystals and closed families A crystal A is a highest-weight crystal with highest-weight λ ∈ Qg if there exists an element aλ ∈ A such that • wt(aλ ) = λ, • e˜i aλ = 0 for any i ∈ I , • any element of A can be obtained from aλ by successive applications of the operators f˜i . Consider a family of highest-weight normal crystals {A (λ)}λ∈J labeled by a set J ⊂ Qg . (The highest weight of A (λ) is λ.) The family {A (λ)}λ∈J is called strictly closed (with respect to tensor products) if the tensor product of any two members of the family is isomorphic to a direct sum of members of the family: M
A (µ1 ) ⊗ A (µ2 ) = U (µ1 , µ2 ), λ ⊗ A (λ), λ∈J

where U ((µ1 , µ2 ), λ) is a set equipped with the trivial crystal structure. The family {A (λ)}λ∈J is called closed if the tensor product A (µ1 ) ⊗ A (µ2 ) of any two members of the family contains Aµ1 +µ2 as a direct summand. Any strictly closed family is closed. The converse is also true, as a corollary of Theorem 1.5. Let Qg+ ⊂ Qg be the set of highest weights of integrable highest-weight modules of g. (In the reductive case a module is called integrable if it is derived from a polynomial representation of the corresponding connected simply connected reductive group; in the Kac-Moody case the highest weight should be a positive linear combination of the fundamental weights.) The original motivation for the introduction of crystals was the discovery by Kashiwara [K2] and Lusztig [L1] of canonical (or crystal) bases in integrable highest-weight modules of a (quantum) Kac-Moody algebra. These bases have many favorable properties, one of which is that as sets they are equipped with a crystal structure. In other words, to each irreducible integrable highest-weight module L(λ) corresponds a normal crystal L (λ) (crystal of the canonical basis). In this way one obtains a strictly closed family of crystals {L (λ)}λ∈Qg+ , satisfying the following two properties:

486 • •

ANTON MALKIN

the cardinal of L (λ) is equal to the dimension of L(λ); one has the following tensor product decompositions for g-modules and gcrystals: M
L(µ1 ) ⊗ L(µ2 ) = C (µ1 , µ2 ), λ ⊗ L(λ), λ∈Qg+

L (µ1 ) ⊗ L (µ2 ) =

M


C (µ1 , µ2 ), λ ⊗ L (λ),

λ∈Qg+

where the cardinal of the set (trivial crystal) C ((µ1 , µ2 ), λ) is equal to the dimension of the linear space (trivial g-module) C((µ1 , µ2 ), λ). The aim of this paper is to construct another strictly closed family of g-crystals {M (λ)}λ∈Qg+ (for g being gl N or a symmetric Kac-Moody algebra), using geometry associated to g. The following crucial theorem ensures that this family is isomorphic to the family {L (λ)}λ∈Qg+ of crystals of canonical bases. (Two families of crystals labeled by the same index set are called isomorphic if the corresponding members of the families are isomorphic as crystals.) 1.5. Uniqueness theorem THEOREM

There exists a unique (up to an isomorphism) closed family of g-crystals labeled Qg+ . Proof For the proof in the Kac-Moody case, see [J, Prop. 6.4.21]. The statement for a reductive g easily follows from the statement for the factor of g by its center.

2. Nakajima varieties and tensor product varieties 2.1. Oriented graphs and path algebras Let I be the set of vertices of the Dynkin graph of a simple simply laced Lie algebra g. Let H be the set of pairs consisting of an edge of the Dynkin graph of g and an orientation of this edge. The target (resp., source) vertex of h ∈ H is denoted by In(h) (resp., Out(h)). Thus (I, H ) is an oriented graph. (Note that it has twice as many edges as the Dynkin graph of g.) For h ∈ H , let h¯ be the same edge but with ¯ = Out(h) and opposite orientation (i.e., h¯ is the unique element of H such that In(h) ¯ Out(h) = In(h)). Let X be a symmetric (I × I )-matrix uniquely defined by the following equation:

TENSOR PRODUCT VARIETIES AND CRYSTALS

hX v, ui =

X

487

vIn(h) u Out(h)

(2.1.1)

h∈H

for any v, u ∈ Z[I ]. Note that X = 2 Id −A, where A is the Cartan matrix of g. Let F be the path algebra of the oriented graph (I, H ) over C. Fix a function ¯ = 0 for any h ∈ H . Let ε : H → C∗ such that ε(h) + ε(h) X θi = ε(h)h h¯ ∈ F , (2.1.2) h∈H In(h)=i

and let F0 be the factor algebra of F by the two-sided ideal generated by elements θi for all i ∈ I . The algebra F0 is called the preprojective algebra. It was introduced by I. Gelfand and V. Ponomarev. Note that an F -module is just a Z[I ]-graded C-linear space V together with a collection of linear maps x = {x h ∈ HomC (VOut(h) , VIn(h) )}h∈H . This F -module is denoted by (V, x). It is always assumed below that V is finite-dimensional. Given a finite-dimensional Z[I ]-graded C-linear space V , a set of linear maps x = {x h ∈ HomC (VOut(h) , VIn(h) )}h∈H such that (V, x) is an F -module is called a representation of F in V . Let (V, x) be an F -module, and let W be a Z[I ]-graded subspace of V such that x h WOut(h) ⊂ WIn(h) for any h ∈ H . Then (W, x|W ) (resp., (V /W, x|V /W )) is an F -submodule (resp., factor module) of (V, x). Sometimes just W (resp., V /W ) is called a submodule (resp., factor module) because the representation of F is uniquely defined by the restriction. P An F -module (V, x) is also an F0 -module if and only if h∈H ε(h)x h x h¯ = 0 In(h)=i

for any i ∈ I . A k-tuple h 1 , h 2 , . . . , h k of elements of H is called a path of length k if In(h i ) = Out(h i−1 ) for i = 2, . . . , k. An F -module (V, x) is called nilpotent if there exists N ∈ Z>0 such that x h 1 x h 2 · · · x h N = 0 for any path h 1 , h 2 , . . . , h N of length N . The following proposition is due to Lusztig [L1]. PROPOSITION

One has the following. (2.1.3) An F -module (V, x) is nilpotent if and only if x h 1 x h 2 · · · x h | dim V| = 0 (i.e., any path of length | dim V | is in the kernel of the representation x). (2.1.4) Any F0 -module is nilpotent as an F -module. Proof The “if” part of (2.1.3) follows from the definition of a nilpotent F -module. The “only if” part of (2.1.3) is proven in [L1, Lem. 1.8].

488

ANTON MALKIN

(2.1.4) is proven in [L1, Sec. 12]. Recall that the Dynkin graph is assumed to be of finite (ADE) type. 2.2. Quiver varieties Quiver varieties have been introduced by Nakajima [N1], [N2] as a generalization of the moduli space of Yang-Mills instantons on an asymptotically locally Euclidean (ALE) space (cf. [KN]). On the other hand, they also generalize the Spaltenstein variety for GL(N ) (the variety of all parabolic subgroups P such that a given unipotent element u ∈ GL(N ) belongs to the unipotent radical of P). Nakajima has shown in [N1], [N2], and [N3] that the quiver varieties play the same role in the representation theory of (quantum) Kac-Moody algebras as Spaltenstein varieties do in the representation theory of GL(N ) (cf. [G], [KL]). Let D, V be finite-dimensional Z[I ]-graded C-linear spaces. An ADHM datum∗ (on D, V ) is a triple (x, p, q), where x = {x h }h∈H is a representation of F in V , p ∈ HomC I (D, V ) is a Z[I ]-graded C-linear map from D to V , q ∈ HomC I (V, D) is a Z[I ]-graded C-linear map from V to D, and x, p, q satisfy the following equation (in EndC Vi ): X ε(h)x h x h¯ − pi qi = 0 (2.2.1) h∈H In(h)=i

for all i ∈ I . The set of all ADHM data on D, V form an affine variety denoted by 3 D,V . Let D = {0}. Then p = q = 0, and x gives a representation of the preprojective algebra F0 in V . The variety 3{0},V is denoted simply by 3V , and an element (x, 0, 0) of 3V is usually written simply as x. Thus 3V is the variety of all representations of the preprojective algebra F0 in V . It follows from (2.1.4) that if x ∈ 3V , then (V, x) is a nilpotent F -module. The variety 3V has been introduced by Lusztig [L1, Sec. 12]. He has also defined an open subset δ 3V of 3V given by n  M  o δ 3V = x ∈ 3V dim Coker x h ≤ δi for any i ∈ I , (2.2.2) h∈H In(h)=i

where δ ∈ Z≥0 [I ]. The following proposition is proven in [L1, Sec. 12]. PROPOSITION

Let v = dim V . Then (2.2.3) δ 3V is either empty or has pure dimension (1/2)hX v, vi, (2.2.4) 3V has pure dimension (1/2)hX v, vi. ∗ ADHM

stands for M. Atiyah, V. Drinfeld, N. Hitchin, and Yu. Manin [AHDM].

TENSOR PRODUCT VARIETIES AND CRYSTALS

489

2.3. Stability and G V -action Let (x, p, q) ∈ 3 D,V , and let E be a graded subspace of V . Then E (resp., E) denotes the smallest F -submodule of (V, x) containing E (resp., the largest F -submodule of (V, x) contained in E). Note that the triple (x|q −1 (0) , 0, 0) (resp., (x|V / p(D) , 0, 0)) belongs to 3q −1 (0) (resp., 3V / p(D) ). Nakajima [N1], [N2] has defined open (possibly empty) subsets of the variety 3 D,V given as sets of points satisfying some stability conditions. Two examples of such subsets are 3sD,V (the set of stable points) and 3∗s D,V (the set of ∗-stable points). The set 3sD,V (resp., 3∗s ) is the set of all triples (x, p, q) ∈ 3 D,V D,V s,∗s s ∗s −1 such that p(D) = V (resp., q (0) = 0). Let 3 D,V = 3 D,V ∩ 3 D,V . Q Let G V = AutC I V = i∈I GL(Vi ) be the group of graded automorphisms of V . The group G V acts on 3V and 3 D,V as follows: (g, x) → x g ,
g, (x, p, q) → (x g , p g , q g ), where g ∈ G V , x ∈ 3V (resp., (x, p, q) ∈ 3 D,V ), and g

−1 x h = gIn(h) x h gOut(h) ,

g

pi = gi pi ,

g

qi = qi gi−1 ,

for h ∈ H , i ∈ I . s,∗s The subsets 3sD,V , 3∗s D,V , and 3 D,V are preserved by the G V -action. The following proposition due to Nakajima and W. Crawley-Boevey explains the importance of the open sets 3sD,V and 3∗s D,V . PROPOSITION

Let d = dim D, v = dim V . Then (2.3.1) 3sD,V is empty or an irreducible smooth variety of dimension dim 3sD,V = hX v, vi + 2hd, vi − hv, vi; (2.3.2) 3∗s D,V is empty or an irreducible smooth variety of dimension dim 3∗s D,V = hX v, vi + 2hd, vi − hv, vi; (2.3.3) 3s,∗s D,V is empty or an irreducible smooth variety of dimension dim 3s,∗s D,V = hX v, vi + 2hd, vi − hv, vi; (2.3.4) 3s,∗s D,V is nonempty if and only if d − 2v + X v ∈ Z≥0 [I ]; s,∗s (2.3.5) G V -action is free on 3sD,V , 3∗s D,V , and 3 D,V . See (2.1.1) for the definition of the matrix X .

490

ANTON MALKIN

Proof The smoothness part of (2.3.2) and statement (2.3.5) in the case of 3∗s D,V are proven by Nakajima [N2, Lem. 3.10]. The fact that 3∗s is connected is proven by CrawleyD,V Boevey ([CB, remarks after the introduction]). The proof of (2.3.1) is analogous to the one of (2.3.2), or one can deduce the former from the latter by transposing (x, p, q). Either argument also proves the part of (2.3.5) concerning 3sD,V . Statement (2.3.3) (and the corresponding part of (2.3.5)) follows from the definis,∗s s ∗s tion of 3s,∗s D,V (3 D,V = 3 D,V ∩ 3 D,V ), (2.3.1), and (2.3.2). Statement (2.3.4) is proven by Nakajima [N2, Secs. 10.5 – 10.9] (see also [L6, Prop. 1.11]). It follows from (2.3.5) and (2.3.1) that MsD,V = 3sD,V /G V is naturally an algebraic variety of dimension dim MsD,V = hX v, vi + 2hd, vi − 2hv, vi, and the natural projection 3sD,V → MsD,V is a principal G V -bundle. The same holds s,∗s s,∗s ∗s s ∗s for M∗s D,V = 3 D,V /G V and M D,V = 3 D,V /G V . The varieties M D,V , M D,V , and s,∗s M D,V are called quiver varieties (cf. [N1], [N2]). s,∗s Since MsD,V , M∗s D,V , and M D,V are G V -orbit spaces, they do not depend on V as long as dim V = v is fixed. Also, if dim D = dim D 0 , the above varieties are isomorphic (noncanonically) to the corresponding varieties defined using D 0 instead of D. As D never varies, the following notation is often used (cf. [N1], [N2]): s,∗s (d, v) = Ms,∗s . Ms (d, v) = MsD,V , M∗s (d, v) = M∗s D,V , M D,V 2.4. The variety Ms (d, v0 , v) Lusztig [L6, Sec. 1] has introduced two kinds of subsets of 3 D,V . Namely, given a Z[I ]-graded subspace U ⊂ V , let  3 D,V,U = (x, p, q) ∈ 3 D,V q −1 (0) = U , and given v0 ∈ Z≥0 [I ], let  3 D,V,v0 = (x, p, q) ∈ 3 D,V dim q −1 (0) = v − v0 . The following proposition is proven in [L6, Sec. 1.8]. PROPOSITION

3 D,V,U and 3 D,V,v0 are locally closed in 3 D,V . Let 3sD,V,U = 3 D,V,U ∩ 3sD,V , let 3sD,V,v0 = 3 D,V,v0 ∩ 3sD,V , and let Ms (d, v0 , v) = 3sD,V,v0 /G V .

TENSOR PRODUCT VARIETIES AND CRYSTALS

491

2.5. The set M (d, v0 , v) Let U , T be complimentary Z[I ]-graded subspaces of V . Then one has a natural map (cf. [L6]) γ : 3 D,V,U → 3U × 3∗s D,T given by
γ ((x, p, q)) = x UU , (x T T , p T D , q DT ) , where the notation p T D means the D → T component of the block matrix p : D → U ⊕ T , and, similarly, x UU , x T T , q DT are block components of x and q. The map γ is well defined because for (x, p, q) ∈ 3 D,V,U the subspace U is equal to q −1 (0) (i.e., it is the maximal F -submodule contained in the kernel of q). The following proposition is due to Lusztig [L6, Sec. 1]. PROPOSITION

Let d = dim D, v = dim V , u = dim U , t = dim T = v − u. Then (2.5.1) the map γ is a vector bundle with fibers of dimension hd, ui+hX t, ui−ht, ui; (2.5.2) if 3sD,V,U is nonempty, then γ (3sD,V,U ) = δ 3U × 3s,∗s D,T , where δ = d − 2t + X t (the matrix X is defined in (2.1.1)), and 3sD,V,U is an open dense subset in γ −1 (δ 3U × 3s,∗s D,T ); s (2.5.3) 3 D,V,U is empty or has pure dimension dim 3sD,V,U =

1 hX u, ui + hX t, vi + hd, vi + hd, ti − ht, vi; 2

(2.5.4) 3sD,V,v0 is empty or has pure dimension dim 3sD,V,v0 =

1 1 hX v, vi + hX v0 , v0 i + hd, vi + hd, v0 i − hv0 , v0 i; 2 2

(2.5.5) Ms (d, v0 , v) is empty or has pure dimension dim Ms (d, v0 , v) 1 1 = hX v, vi + hX v0 , v0 i + hd, vi + hd, v0 i − hv0 , v0 i − hv, vi 2 2 1 1 = dim Ms (d, v) + dim Ms,∗s (d, v0 ). 2 2 Proof Statements (2.5.1) and (2.5.2) are proven in [L6, Prop. 1.16] (see the proof of Prop. 2.7 for a similar argument). Statement (2.5.3) follows from (2.5.1), (2.5.2), (2.3.3), and (2.2.4). V Let Grw denote the variety of all Z[I ]-graded subspaces of graded dimension w in V . It is a G V -homogeneous variety with connected stabilizer of a point, and

492

ANTON MALKIN

V V dim Grw = hw, v − wi. The map 3sD,V,v0 → Grv−v given by (x, p, q) → q −1 (0) is 0 V V a locally trivial fibration over Grv−v0 with the fiber over U ∈ Grv−v equal to 3sD,V,U . 0 Now (2.5.4) follows from (2.5.3). Statement (2.5.5) follows from (2.5.4) and (2.3.5).

Statement (2.5.5) also follows from [N2, proof of Th. 7.2]. Let M (d, v0 , v) be the set of irreducible components of the variety Ms (d, v0 , v). Since 3sD,V,v0 is the total space of a principal G V -bundle over Ms (d, v0 , v), while V 3sD,V,U (where dim U = v − v0 ) is a fiber of the fibration 3sD,V,v0 → Grv−v over 0 a simply connected homogeneous space, one has natural bijections between the sets of irreducible components of 3sD,V,v0 and 3sD,V,U and the set M (d, v0 , v). Abusing notation, the former two sets (of irreducible components) are also denoted by M (d, v0 , v). F Let M (d, v0 ) = v∈Z[I ] M (d, v0 , v). In Section 3.8 the set M (d, v0 ) is equipped with the structure of a g-crystal, which is shown (in Sec. 3.10) to coincide with the crystal of the canonical basis of a highest-weight representation of g. 2.6. Tensor product varieties and multiplicity varieties Given an n-tuple v of elements of Z[I ], vk denotes the kth component of v. Thus v = (v1 , . . . , vn ), vk ∈ Z[I ] (1 ≤ k ≤ n), vik ∈ Z (i ∈ I ). Let D be a Z[I ]-graded C-linear space, and let D = {0 = D0 ⊂ D1 ⊂ D2 ⊂ · · · ⊂ Dn = D} be a partial Z[I ]-graded flag in D. Then dim D denotes an n-tuple of elements of Z[I ] given as (dim D)k = dim Dk − dim Dk−1 (for 1 ≤ k ≤ n). Recall that if (x, p, q) ∈ 3 D,V and if E is a Z[I ]-graded subspace of V , then E (resp., E) denotes the smallest F -submodule of (V, x) containing E (resp., the largest F -submodule of (V, x) contained in E). Let D, V be Z[I ]-graded C-linear spaces, and let D = {0 = D0 ⊂ D1 ⊂ D2 ⊂ · · · ⊂ Dn = D} be a partial Z[I ]-graded flag in D. Then n 5sD,D,V denotes the set of all (x, p, q) ∈ 3sD,V such that p(Dk ) ⊂ q −1 (Dk )

(2.6.1)

for all k = 1, . . . , n. Note that if (x, p, q) ∈ n 5sD,D,V , then p(D) = V (because (x, p, q) ∈ 3sD,V ), and the triple
x| p(Dk )/( p(Dk )∩q −1 (Dk−1 )) , p|Dk mod q −1 (Dk−1 ), q| p(Dk ) mod Dk−1 belongs to 3s,∗s k

D /Dk−1 , p(Dk )/( p(Dk )∩q −1 (Dk−1 ))

for all k = 1, . . . , n.

TENSOR PRODUCT VARIETIES AND CRYSTALS

493

Let v be an n-tuple of elements of Z≥0 [I ]. Then 
n s 5 D,D,V,v = (x, p, q) ∈ n 5sD,D,V dim p(Dk )/( p(Dk ) ∩ q −1 (Dk−1 )) = vk . Let v˜ be another n-tuple of elements of Z≥0 [I ] such that dim V . Then n

Pn

k

k=1 v

+

Pn

k

˜ k=1 v

=

5sD,D,V,v,˜v 
= (x, p, q) ∈ n 5sD,D,V,v dim ( p(Dk ) ∩ q −1 (Dk−1 ))/ p(Dk−1 ) = v˜ k .

˜ 1 ⊂ V1 ⊂ V ˜2 ⊂ ··· ⊂ V ˜ n ⊂ Vn = V ) be a 2n-step Let V = (0 = V0 ⊂ V Z[I ]-graded partial flag in V . Then  n s ˜k . 5 D,D,V,V = (x, p, q) ∈ n 5sD,D,V p(Dk ) = Vk , q −1 (Dk−1 ) ∩ p(Dk ) = V ˜ k , v˜ k = Note that n 5sD,D,V,V ⊂ n 5sD,D,V,v,˜v , where vk = dim Vk − dim V k k−1 ˜ − dim V . dim V The proof of the following proposition is analogous to the proof of Proposition 2.4. PROPOSITION

The sets n 5sD,D,V,v , n 5sD,D,V,v,˜v , and n 5sD,D,V,V are locally closed in 3 D,V . s,∗s n s Let n 5s,∗s D,D,V,v = 5 D,D,V,v ∩ 3 D,V (a locally closed subset of 3 D,V ). The sets s,∗s n 5s n D,D,V,v and 5 D,D,V,v are invariant under the action of G V , and, moreover, this action is free (because of (2.3.5)). Hence n T D,D,V,v = n 5sD,D,V,v /G V and nS n s,∗s D,D,V,v = 5 D,D,V,v /G V are naturally quasi-projective varieties. The variety n T D,D,V,v is called a tensor product variety because of its role in the geometric description of the tensor product of finite-dimensional representations of g. In particular, it is shown below (see (3.10.3)) that the set of irreducible components of n T D,D,V,v is in a bijection with a weight subset of the crystal of the canonical basis of a product of n representations of g. The variety n S D,D,V,v is called a multiplicity variety because, as proven below (see (3.10.2)), the number of its irreducible components is equal to the multiplicity of a certain representation of g in a tensor product of n representations. The varieties n T D,D,V,v and n S D,D,V,v do not depend on the choice of V as long as dim V = v is fixed, and they do not depend up to a noncanonical isomorphism on D and D. As D and D never vary in this paper, a simpler notation is used: n T(d, d, v, v) = n T n n D,D,V,v and S(d, d, v, v) = S D,D,V,v , where d = dim D, d = dim D (cf. the end of Sec. 2.3).

494

ANTON MALKIN

2.7. Inductive construction of tensor product varieties ˜ 1 ⊂ V1 ⊂ V ˜2 ⊂ ··· ⊂ V ˜ n ⊂ Vn = V ) be a 2n-step Z[I ]Let V = (0 = V0 ⊂ V 0 graded partial flag in V , and let D = (0 = D ⊂ D1 ⊂ · · · ⊂ Dn = D) be an n-step Z[I ]-graded partial flag in D. Assume n ≥ 2, and choose an integer k such that 0 < k < n. Let U (resp., C) be a Z[I ]-graded subspace in V (resp., in D) complimentary ˜ 1 ⊂ V1 ⊂ V ˜2 ⊂ ··· ⊂ V ˜ k ⊂ Vk ) (resp., to Vk (resp., Dk ). Let V0 = (0 = V0 ⊂ V D0 = (0 = D0 ⊂ D1 ⊂ · · · ⊂ Dk )) be a 2k-step Z[I ]-graded partial flag in Vk (resp., a k-step Z[I ]-graded partial flag in Dk ), obtained by restricting the flag V (resp., D). ˜ 1 ⊂ U1 ⊂ U ˜2 ⊂ ··· ⊂ U ˜ n−k ⊂ Un−k = U ) (resp., Similarly, let U = (0 = U0 ⊂ U 0 1 n−k C = (0 = D ⊂ D ⊂ · · · ⊂ D = D)) be a 2(n − k)-step Z[I ]-graded partial flag in U (resp., an (n − k)-step Z[I ]-graded partial flag in C), obtained by taking intersections of the subspaces of the flag V (resp., D) with U (resp., C). One has a regular map ρ2 :

n

s

s

5sD,D,V,V → k 5Dk ,D0 ,Vk ,V0 × n−k 5C,C,U,U

given by ρ2 (x, p, q) = (x V

k Vk

k Dk

, pV

, qD

k Vk


), (x UU , pU C , q CU ) ,

where, for example, pU C is the C → U component of the block matrix p : D = Dk ⊕ C → Vk ⊕ U = V , and the other maps in the right-hand side are defined similarly. PROPOSITION

The map ρ2 is a vector bundle with fibers of dimension hX u, v − ui + hc, v − ui + hd − c, ui − hu, v − ui, where v = dim V , d = dim D, u = dim U , c = dim C. Proof k k k k k k The fiber of the map ρ2 over a point ((x V V , p V D , q D V ), (x UU , pU C , q CU )) in k 5s n−k 5s Dk ,D0 ,Vk ,V0 × C,C,U,U consists of all linear maps x hV

kU

∈ HomC (UOut(h) , VkIn(h) ),

kC

∈ HomC (Ci , Vik ),

kU

∈ HomC (Ui , Dik )

piV qiD

(where i ∈ I , h ∈ H ), subject to the linear equations X k k k k Vk C CU Vk Dk Dk U qi = 0 ε(h)(x hV V x hV¯ U + x hV U x hUU ¯ ) − pi qi − pi h∈H In(h)=i

TENSOR PRODUCT VARIETIES AND CRYSTALS

495

for any i ∈ I . Now the proposition follows from the fact that the linear map M

 M  M  HomC (UOut(h) , VkIn(h) ) ⊕ HomC (Ci , Vik ) ⊕ HomC (Ui , Dik )

h∈H

i∈I

i∈I

→

M

HomC (Ui , Vik ) (2.7.1)

i∈I

given by X  Vk U Vk C Dk U k k k k Vk C CU Vk Dk Dk U x , p ,q → ε(h)(x hV V x hV¯ U + x hV U x hUU qi ¯ ) − pi qi − pi h∈H In(h)=i

L is surjective. To prove this, let a ∈ i∈I HomC (Vik , Ui ) be orthogonal to the image of map (2.7.1) with respect to the tr pairing; that is, let X k k k k Vk C CU Vk Dk Dk U ε(h) tr(x hV V x hV¯ U ai + x hV U x hUU qi ai = 0 ¯ ai ) − tr pi qi ai − tr pi h∈H In(h)=i k

k

k

for any x V U , p V C , q D U , and any i ∈ I . It follows that aIn(h) x hV

k Vk

= x hUU aOut(h) ,

qiCU ai = 0, ai piV

k Dk

= 0, k k

for any h ∈ H , i ∈ I . In particular, the kernel of a contains the image of p V D and k k k k k k k k is x V V -invariant. Since (x V V , p V D , q D V ) is stable, it follows that a = 0, and hence map (2.7.1) is surjective. The proposition is proven. 2.8. Dimensions of tensor product and multiplicity varieties The following proposition gives the dimensions of various varieties defined above. PROPOSITION

Let d = dim D, v = dim V . Then (2.8.1) n 5sD,D,V,V is empty or has pure dimension 1 1 dim n 5sD,D,V,V = hX v, vi + hd, vi − hv, vi 2 2 n   X 1 1 1 + hX vs , vs i + hds , vs i − hvs , vs i + h˜vs , v˜ s i , 2 2 2 s=1

˜ k , v˜ k = dim V ˜ k − dim Vk−1 ; where vk = dim Vk − dim V

496

ANTON MALKIN

(2.8.2) n 5sD,D,V,v,˜v is empty or has pure dimension 1 dim n 5sD,D,V,v,˜v = hX v, vi + hd, vi 2 n   X 1 + hX vs , vs i + hds , vs i − hvs , vs i ; 2 s=1

(2.8.3)

n 5s D,D,V,v

is empty or has pure dimension

1 dim n 5sD,D,V,v = hX v, vi + hd, vi 2 n   X 1 + hX vs , vs i + hds , vs i − hvs , vs i ; 2 s=1

(2.8.4)

n 5s,∗s D,D,V,v

is empty or has pure dimension

1 dim n 5s,∗s D,D,V,v = hX v, vi + hd, vi 2 n   X 1 + hX vs , vs i + hds , vs i − hvs , vs i ; 2 s=1

(2.8.5) the tensor product variety n T(d, d, v, v) is empty or has pure dimension dim n T(d, d, v, v) =

1 hX v, vi + hd, vi − hv, vi 2 n   X 1 + hX vs , vs i + hds , vs i − hvs , vs i 2 s=1

n  X 1 s dim M (d, v) + dim Ms,∗s (ds , vs ) ; = 2 s=1

(2.8.6) the multiplicity variety

n S(d, d, v, v)

dim n S(d, d, v, v) =

is empty or has pure dimension

1 hX v, vi + hd, vi − hv, vi 2 n   X 1 + hX vs , vs i + hds , vs i − hvs , vs i 2 s=1

n  X 1 = dim Ms,∗s (d, v) + dim Ms,∗s (ds , vs ) . 2 s=1

Proof Statement (2.8.1) follows by induction in n using Proposition 2.7. The base of the induction is provided by (2.5.3).

TENSOR PRODUCT VARIETIES AND CRYSTALS

497

Statement (2.8.2) follows from (2.8.1) using the fact that n 5sD,D,V,v,˜v is a fibration with fibers isomorphic to n 5sD,D,V,V over the variety of graded 2n-step partial flags in V with dimensions of the subfactors given by v and v˜ . The dimension of this flag variety is equal to n X
 1 hv, vi − hvs , vs i + h˜vs , v˜ s i . 2 s=1

Statement (2.8.3) follows from the fact that n 5sD,D,V,v is a union of a finite number of locally closed subsets n 5sD,D,V,v,˜v (for different v˜ ) having identical dimensions (cf. (2.8.2)). n s Statement (2.8.4) follows from the fact that n 5s,∗s D,D,V,v is open in 5 D,D,V,v and from (2.8.3). To prove (2.8.5) (resp., (2.8.6)), note that n T(d, d, v, v) = n 5sD,D,V,v /G V (resp., n S(d, d, v, v) = n 5s,∗s D,D,V,v /G V ), and note that the action of the group G V on n 5s n 5s,∗s (resp., D,D,V,v D,D,V,v ) is free. Now (2.8.5) (resp., (2.8.6)) follows from (2.8.3) (resp., (2.8.4)). 2.9. Irreducible components of tensor product varieties Let n T (d, d, v, v) (resp., n S (d, d, v, v)) denote the set of irreducible components of the tensor product variety n T(d, d, v, v) (resp., the multiplicity variety n S(d, d, v, v)). Since n 5sD,D,V,v is the total space of a principal G V -bundle over n T(d, d, v, v), the set of irreducible components of n 5sD,D,V,v can be naturally identified with n T (d, d, v, v). On the other hand, n 5s D,D,V,v is a union of locally closed subsets n 5s ˜ ) having the same dimension (cf. Prop. 2.8). Hence D,D,V,v,˜v (for different v G n n T (d, d, v, v) = T (d, d, v, v, v˜ ), v˜

P P where v˜ ranges over all n-tuples of elements of Z[I ] such that ns=1 vs + ns=1 v˜ s = v, and n T (d, d, v, v, v˜ ) denotes the set of irreducible components of n 5sD,D,V,v,˜v . The variety n 5sD,D,V,v,˜v is a locally trivial fibration over the (simply connected G V homogeneous) variety of all graded 2n-step partial flags in V with dimensions of the subfactors given by v and v˜ (as in Sec. 2.6). A fiber of this fibration is isomorphic to n 5s D,D,V,V , where V is a point of the base (a 2n-step partial flag). It follows that the set of irreducible components of n 5sD,D,V,V can be naturally identified with the set n T (d, d, v, v, v ˜ ). To summarize, n T (d, d, v, v) is the set of irreducible components s of n 5 D,D,V,v and n T(d, d, v, v), n T (d, d, v, v, v˜ ) is the set of irreducible compoF nents of n 5sD,D,V,v,˜v and n 5sD,D,V,V , and n T (d, d, v, v) = v˜ n T (d, d, v, v, v˜ ). Similarly, n S (d, d, v, v) is the set of irreducible components of n S(d, d, v, v) and n 5s,∗s D,D,V,v .

498

ANTON MALKIN

Finally, let n T (d, d, v) =

F

v∈Z≥0 [I ]

nT

(d, d, v, v).

2.10. The first bijection for a tensor product variety Let 1 < k < n. The vector bundle ρ2 introduced in Section 2.7 induces a bijection of the sets of irreducible components αk,n−k :

n

∼

T (d, d, v, v, v˜ ) − → k T (d 0 , d0 , v 0 , v0 , v˜ 0 ) × n−k T (d 00 , d00 , v 00 , v00 , v˜ 00 ),

where d, d 0 , d 00 , v, v 0 , v 00 ∈ Z≥0 [I ], d = d 0 + d 00 , v = v 0 + v 00 , d, v, v˜ are ntuples of elements of Z≥0 [I ], d0 , v0 , v˜ 0 are k-tuples of elements of Z≥0 [I ], d00 , v00 , v˜ 00 are (n − k)-tuples of elements of Z≥0 [I ], d = (d0 , d00 ), v = (v0 , v00 ), v˜ = (˜v0 , v˜ 00 ), P P 00s ˜ 00s ). Taking union over v˜ , one obtains a v 0 = ks=1 (v0s + v˜ 0s ), v 00 = n−k s=1 (v + v bijection (denoted again by αk,n−k ) G ∼ k αk,n−k : n T (d, d, v, v) − → T (d 0 , d0 , v 0 , v0 ) × n−k T (d 00 , d00 , v 00 , v00 ). v 0 ,v 00 ∈Z≥0 [I ] v 0 +v 00 =v

Generalizing the map ρ2 (cf. Sec. 2.7), one can consider a map (given by restriction of x, p, and q) ρ3 :

n

5sD,D,V,V s

s

s

→ k 5Dk ,D0 ,Vk ,V0 × l−k 5Dl /Dk ,D00 ,Vl /Vk ,V00 × n−l 5 D/Dl ,D000 ,V /Vl ,V000 , where 1 < k < l < n, D0 (resp., D00 , D000 ) is the k-step partial flag in Dk (resp., (l − k)-step partial flag in Dl /Dk , (n − l)-step partial flag in D/Dl ) induced by the n-step flag D in D, and, similarly, V0 (resp., V00 , V000 ) is the 2k-step partial flag in Vk (resp., 2(l − k)-step partial flag in Vl /Vk , 2(n − l)-step partial flag in V /Vl ) induced by the 2n-step flag V in V . The map ρ3 can be represented in two ways as a composition of two maps ρ2 : n 5s D,D,V,V KK

ρ2

ρ2

s

l5 ¯¯ l ,V ¯¯ Dl ,D,V / s n−l × 5 D/Dl ,D000 ,V /Vl ,V000

KK KK KK KK KK KK ρ3 KKK KK KK KK KK %

 k 5s Dk ,D0 ,Vk ,V0 s ×n−k 5 D/Dk ,D,V ¯ /Vk ,V ¯

Id ×ρ2

ρ2 ×Id

 k 5s Dk ,D0 ,Vk ,V0 / l−k 5s l k 00 l k 00 × D /D ,D ,V /V ,V s ×n−l 5 D/Dl ,D000 ,V /Vl ,V000

(2.10.1)

¯¯ is the partial flag in D/Dk (resp., partial flag in Dl ) induced by the ¯ (resp., D) where D ¯¯ is the partial flag in V /Vk (resp., partial flag in ¯ (resp., V) flag D in D, and, similarly, V

TENSOR PRODUCT VARIETIES AND CRYSTALS

499

Vl ) induced by the flag V in V . It follows from the commutativity of diagram (2.10.1) and Proposition 2.7 that ρ3 is a locally trivial fibration with a smooth connected fiber and, therefore, induces a bijection
αk,l−k,n−l : n T (d 0 + d 00 + d 000 ), (d0 , d00 , d000 ), v, (v0 , v00 , v000 ) G ∼ k − → T (d 0 , d0 , v 0 , v0 ) × l−k T (d 00 , d00 , v 00 , v00 ) v 0 ,v 00 ,v 000 ∈Z≥0 [I ] v 0 +v 00 +v 000 =v n−l 000 000

T (d , d , v 000 , v000 ).

×

Moreover, the commutativity of diagram (2.10.1) implies that αk,l−k,n−l = (αk,l−k × Id) ◦ αl,n−l = (Id ×αl−k,n−l ) ◦ αk,n−k . One can consider analogues of the fibrations ρ2 and ρ3 taking values in the product of any number (less than or equal to n) of varieties 5s . These maps can be represented as compositions of the maps ρ2 in several ways (cf. (2.10.1)); hence they are fibrations with smooth connected fibers and induce isomorphisms of the sets of irreducible components. The most important case is the map ρn that takes values in the s s product of n varieties 1 5 . (Recall that the variety 1 5 is the same as the variety 3s .) ˆ s (resp., V ˆ s ) be a Z[I ]-graded subspace in Ds complimentary to More precisely, let D s−1 D (resp., a Z[I ]-graded subspace in Vs complimentary to Vs−1 ), and let ρn :

n

5sD,D,V,V → 3sˆ 1

ˆ1 D ,V

× · · · × 3sˆ n

ˆn D ,V

be a regular map given by ˆ 1V ˆ1

ρn (x, p, q) = (x V

ˆ 1D ˆ1

, pV

ˆ 1V ˆ1

, qD

ˆ nV ˆn

), . . . , (x V

ˆ nD ˆn

, pV

ˆ nV ˆn

, qD


) .

The map ρn can be represented (in many ways) as a composition of maps ρ2 . Therefore it is a fibration with smooth connected fibers, and it induces a bijection αn :

n

∼

T (d, d, v, v, v˜ ) − → M (d1 , v1 , v1 + v˜ 1 ) × · · · × M (dn , vn , vn + v˜ n )

or, after taking union over v˜ , αn :

n

∼

G

T (d, d, v, v) − →

M (d1 , v1 , u1 ) × · · · × M (dn , vn , un ).

u1 ,...,un ∈Z≥0 [I ] u1 +···+un =v

Finally, union over v gives a bijection αn :

n

∼

T (d, d, v) − → M (d1 , v1 ) × · · · × M (dn , vn ).

(2.10.2)

500

ANTON MALKIN

Recall that M (d, v, u) denotes the set of irreducible components of a certain locally closed subset of a quiver variety (cf. Sec. 2.5). The bijection αn justifies the name tensor product variety given to n T(d, d, v, v) since it is known that the set M (d, v) can be equipped with the structure of a g-crystal, which is isomorphic to the crystal of the canonical basis of an irreducible finite-dimensional representation of g (see Sec. 2.5 for references and Sec. 3.8 for a proof). 2.11. The multiplicity variety for two multiples Let D = D 1 ⊕ D 2 , and let D = (0 ⊂ D 1 ⊂ D) (a flag in D). Similarly, let V = V 1 ⊕ U ⊕ V 2 , and let V = (0 ⊂ V 1 ⊂ V 1 ⊕ U ⊂ V ). Then one has the following map: σ2 :

2

s,∗s

5 D,D,V,V → 3s,∗s × 3U × 3s,∗s D 1 ,V 1 D 2 ,V 2

(2.11.1)

given by σ2 (x, p, q) = (x V

1V 1

, pV

1 D1

, qD

1V 1

), x UU , (x V

2V 2

, pV

2 D2

, qD

2V 2


) .

PROPOSITION Let v 1 = dim V 1 ,

v 2 = dim V 2 , u = dim U , d 1 = dim D 1 , d 2 = dim D 2 , v = dim V = + u, d = dim D = d 1 + d 2 . Then 0 × 3s,∗s , where 30 is an open subset of (2.11.2) the image of σ2 is 3s,∗s × 3U U D 1 ,V 1 D 2 ,V 2 3U , (2.11.3) σ2 is a fibration with smooth connected fibers of dimension v1

+ v2


1 hX v, vi − hX v 1 , v 1 i − hX u, ui − hX v 2 , v 2 i 2 + hd, ui − hd 1 , v 1 i − hd 2 , v 2 i
1 + hv, vi − hv 1 , v 1 i − hu, ui − hv 2 , v 2 i . 2 Proof The proof is analogous to the proofs of Propositions 2.5 and 2.7. Namely, given 1 1 1 1 1 1 2 2 2 2 2 2
(x V V , p V D , q D V ), x UU , (x V V , p V D , q D V ) ∈ 3s,∗s × 3U × 3s,∗s , D 1 ,V 1 D 2 ,V 2 the fiber of σ2 over this point is a vector bundle σ 0 over an affine space consisting of all linear maps 1 2 2 1 x V U , x U V , pU D , q D U

TENSOR PRODUCT VARIETIES AND CRYSTALS

501

subject to the equations X 1 1 1 1 V 1 D1 D1 U ε(h)(x hV V x hV¯ U + x hV U x hUU qi = 0, ¯ ) − pi h∈H In(h)=i

X

2

2

ε(h)(x hUU x hU¯ V + x hU V x hV¯

2V 2

2

) − piU D qiD

2V 2

= 0.

h∈H In(h)=i

Now (2.11.2) follows from the condition that (x, p, q) ∈ 5s,∗s D,D,V,V . The proof is similar to that of (2.5.2) (cf. [L6, Prop. 1.16]), or one can deduce (2.11.2) from (2.5.2) and its dual using the fact that (x, p, q) ∈ 5s,∗s D,D,V,V if and only if (x (V

1 ⊕U )(V 1 ⊕U )

, p (V

1 ⊕U )D 1

, qD

1 (V 1 ⊕U )

) ∈ 3∗s D 1 ,V 1 ⊕U

and (x (U ⊕V

2 )(U ⊕V 2 )

, p (U ⊕V

2 )D 2

, qD

2 (U ⊕V 2 )

) ∈ 3sD 2 ,U ⊕V 2 .

The fiber of σ 0 over a point ((x V

1V 1

, pV

1 D1

, qD

1V 1

), x UU , (x V

2V 2

, pV

2 D2

, qD

2V 2


1 2 2 1 )), (x V U , x U V , pU D , q D U )

is an affine space of all linear maps xV

1V 2

, pV

1 D2

, qD

1V 2

subject to the equations X 1 1 1 2 1 2 2 2 1 1 1 2 1 2 2 2 ε(h)(x hV V x hV¯ V + x hV V x hV¯ V ) − piV D qiD V − piV D qiD V = 0. h∈H In(h)=i

One has to check that the systems of linear equations used in the proof are not 1 1 1 1 1 1 overdetermined, which follows from the fact that (x V V , p V D , q D V ) ∈ 3s,∗s D 1 ,V 1

and (x V

2V 2

, pV

2 D2

, qD

2V 2

) ∈ 3s,∗s (cf. the proof of Prop. 2.7). D 2 ,V 2

Note that because 3s,∗s D,V is smooth and connected (cf. (2.3.3)), Proposition 2.11 ims,∗s plies that the set of irreducible components of 2 5 D,D,V,V is in a natural bijection with 0 . the set of irreducible components of 3U 2.12. The first bijection for a multiplicity variety 1 k 2 Let 1 < k < n. Consider the variety n 5s,∗s D,D,V,V . Let D = D , and let D be a Z[I ]-graded subspace in D complimentary to Dk . Similarly, let V 1 = Vk , let U be ˜ k+1 complimentary to Vk , and let V 2 be a Z[I ]-graded a Z[I ]-graded subspace in V

502

ANTON MALKIN

˜ k+1 . Let D ¯ = (0 ⊂ D 1 ⊂ D) (a subflag of D), subspace in V complimentary to V 1 1 ¯ let V = (0 ⊂ V ⊂ V ⊕ U ⊂ D) (a subflag of V), let D0 (resp., D00 ) be the kstep flag in D 1 (resp., the (n − k)-step flag in D 2 ) obtained by considering the first k subspaces of D (resp., by taking intersections of the last n − k subspaces of D with D 2 ), and let V0 (resp., V00 ) be the 2k-step flag in V 1 (resp., the 2(n − k)-step flag in V 2 ) obtained by considering the first 2k subspaces of V (resp., by taking intersec2 s,∗s tions of the last 2(n − k) subspaces of V with V 2 ). Then n 5s,∗s ¯ ¯, D,D,V,V ⊂ 5 D,D,V, V k 5s,∗s D 1 ,D0 ,V 1 ,V0

s,∗s

⊂ 3s,∗s , n−k 5 D 2 ,D00 ,V 2 ,V00 ⊂ 3s,∗s , and the map σ2 (cf. (2.11.1)) D 1 ,V 1 D 2 ,V 2 restricts to a fibration (with the same fibers as those of σ2 ) σk,n−k :

n

s,∗s

s,∗s

k 0 n−k 5s,∗s 5 D 2 ,D00 ,V 2 ,V00 . D,D,V,V → 5 D 1 ,D0 ,V 1 ,V0 × 3U ×

Having smooth connected fibers, this fibration induces a bijection ηk,n−k :

n


S d 0 + d 00 , (d0 , d00 ), v, (v0 , v00 ), (˜v0 , v − v 0 − v 00 , v˜ 00 ) ∼

− → k S (d 0 , d0 , v 0 , v0 , v˜ 0 ) × n−k S (d 00 , d00 , v 00 , v00 , v˜ 00 )
× 2 S d 0 + d 00 , (d 0 , d 00 ), v, (v 0 , v 00 ), (0, v − v 0 − v 00 ) , where the last multiple in the right-hand side represents the set of irreducible components of 30 , which is in a natural bijection (induced by the fibration σ2 ; cf. Sec. 2.11) s,∗s with the set of irreducible components of 2 5 D,D,V, ¯ ¯. V 0 00 Taking union over v˜ and v˜ , one obtains a bijection
ηk,n−k : n S d 0 + d 00 , (d0 , d00 ), v, (v0 , v00 ) G ∼ k − → S (d 0 , d0 , v 0 , v0 ) × n−k S (d 00 , d00 , v 00 , v00 ) v 0 ,v 00 ∈Z≥0 [I ]


× 2 S d 0 + d 00 , (d 0 , d 00 ), v, (v 0 , v 00 ) , which is an analogue of the recurrence relation between multiplicities in tensor products of representations of g. As a generalization of the map σk,n−k , one can consider a map σn :

n

5s,∗s D,D,V,V

→ 3s,∗s 1

˜1 D /D0 ,V1 /V

× · · · × 3s,∗s n

˜n D /Dn−1 ,Vn /V

× 3V˜ 1 /V0 × · · · × 3V˜ n /Vn−1

(2.12.1)

defined by restricting x, p, and q. The map σn can be represented (in many ways) as a composition of the maps of type σn,n−k for different n and k (cf. a similar statement for the map ρn in Sec. 2.10). Therefore the image of σn is equal to 3s,∗s 1

˜1 D /D0 ,V1 /V

× · · · × 3s,∗s n

˜n D /Dn−1 ,Vn /V

× X,

TENSOR PRODUCT VARIETIES AND CRYSTALS

503

where X is an open subset of 3V˜ 1 /V0 × · · · × 3V˜ n /Vn−1 , and n 5s,∗s D,D,V,V is an open subset of the total space of a locally trivial fibration σ 0 over the image of σ with a smooth connected fiber, and such that σ 0 restricted to n 5s,∗s D,D,V,V is equal to σ . n In particular, the set S (d, d, v, v) of irreducible components of n 5s,∗s D,D,V,V is in a natural bijection with the set of irreducible components of X . P If nk=1 vk = v, the map σn is a vector bundle σn :

n

s,∗s s,∗s 5s,∗s D,D,V,V → 3D1 /D0 ,V1 /V0 × · · · × 3Dn /Dn−1 ,Vn /Vn−1 ,

which implies the following proposition. PROPOSITION

P Assume that nk=1 vk = v and that 3s,∗s is nonempty for all k = Dk /Dk−1 ,Vk /Vk−1 n 1, . . . , n. Then the set S (d, d, v, v) is a one-element set. 2.13. The second bijection Recall that the tensor product variety n 5sD,D,V,v is a locally closed subset of 3sD,V , and recall that 3sD,V,U denotes the locally closed subset of 3sD,V consisting of all (x, p, q) ∈ 3sD,V such that q −1 (0) = U (cf. Sec. 2.5). Let n 5sD,D,V,U,v = s n 5s D,D,V,v ∩3 D,V,U . Let T be a Z[I ]-graded subspace of V complimentary to U . The n s vector bundle γ : 3 D,V,U → 3U × 3∗s D,T (cf. Prop. 2.5) restricted to 5 D,D,V,U,v s,∗s δ n δ has image equal to 3U × 5 D,D,T,v , where 3U is as in (2.5.2), and, moreover, s n s γ −1 (δ 3U × n 5s,∗s D,D,T,v ) ∩ 3 D,V,U = 5 D,D,V,U,v .

Hence n 5sD,D,V,U,v is open in γ −1 (δ 3U × n 5s,∗s D,D,T,v ). Since fibers of γ have dimension hd, ui − hu, v − ui + hX u, v − ui (cf. Prop. 2.5), it follows that n 5sD,D,V,U,v has pure dimension dimn 5sD,D,V,U,v = dim δ 3U + dim n 5s,∗s D,D,T,v + hd, ui − hu, v − ui + hX u, v − ui = dim n 5sD,D,V,v − hu, v − ui. The set of irreducible components of n 5sD,D,V,U,v is in a natural bijection (induced by the restriction of the vector bundle γ to n 5sD,D,V,U,v ) with the set M (d, v − u, v) × n S (d, d, v − u, v),

where M (d, v − u, v) represents the set of irreducible components of δ 3U , which is in a natural bijection (induced by γ ; cf. Sec. 2.5) with the set of irreducible components of the quiver variety M(d, v − u, v), and n S (d, d, v − u, v) represents the

504

ANTON MALKIN

set of irreducible components of the variety n 5s,∗s D,D,T,v (or the multiplicity variety n S(d, d, v − u, v); cf. Sec. 2.9). Let n 5sD,D,V,u,v be the set of all (x, p, q) ∈ n 5sD,D,V,v such that dim q −1 (0) = u. It is the total space of a fibration over the graded Grassmannian GruV with the fiber over U ∈ GruV equal to n 5sD,D,V,U,v . Hence n 5sD,D,V,u,v has pure dimension dim n 5sD,D,V,u,v = dim n 5sD,D,V,v . In particular, the dimension does not depend on u. Therefore one obtains the following natural bijection of sets of irreducible components: G ∼ n βn : n T (d, d, v, v) − → S (d, d, v0 , v) × M (d, v0 , v). (2.13.1) v0 ∈Z≥0

The bijection βn is an analogue of the direct sum decomposition for the tensor product of n representations of g. In particular, (2.13.1) explains the name multiplicity variety given to n S(d, d, v0 , v). 2.14. The tensor decomposition bijection Let τn be a bijection τn :

G

M (d1 , v1 , u1 ) × · · · × M (dn , vn , un )

u1 ,...,un ∈Z[I ] u1 +···+un =v ∼

− →

G

n

S (d, d, v0 , v) × M (d, v0 , v) (2.14.1)

v0 ∈Z≥0

given by τn = βn ◦ αn−1 . Taking union over v, one obtains a bijection τn :

∼

M (d1 , v1 ) × · · · × M (dn , vn ) − →

G

n

S (d, d, v0 , v) × M (d, v0 ).

v0 ∈Z≥0

(2.14.2) It follows from the definitions of the bijections αn (cf. Sec. 2.10), βn (cf. Sec. 2.13), and ηk,n−k (cf. Sec. 2.12) that −1 τn = (ηk,n−k × τ2 ) ◦ (τk × τn−k ).

(2.14.3)

TENSOR PRODUCT VARIETIES AND CRYSTALS

505

More precisely, the following diagram of bijections is commutative:

M (d1 , v1 ) × · · · × M (dk , vk ) ×M (dk+1 , vk+1 ) × · · · × M (dn , vn )

τn

τk ×τn−k

F v00 ,v000



k S (d 0 , d0 , v 0 , v0 ) × M (d 0 , v 0 ) 0 0 ×n−k S (d 00 , d00 , v000 , v00 ) × M (d 00 , v000 )

Id ×P23 ×Id

F v00 ,v000



k S (d 0 , d0 , v 0 , v0 ) × n−k S (d 00 , d00 , v 00 , v00 ) 0 0 ×M (d 0 , v00 ) × M (d 00 , v000 )

Id × Id ×τ2

F v0 ,v00 ,v000



k S (d 0 , d0 , v 0 , v0 ) × n−k S (d 00 , d00 , v 00 , v00 ) 0 0 ×2 S (d 0 + d 00 , (d 0 , d 00 ), v0 , (v00 , v000 )) × M (d, v0 )

−1 ηk,n−k ×Id

F v0

n S (d 0

 + d 00 , (d0 , d00 ), v0 , (v0 , v00 )) × M (d, v0 )o

(2.14.4) where v0 = (v1 , . . . , vk ), v00 = (vk+1 , . . . , vn ), d0 = (d1 , . . . , dk ), and d00 = (dk+1 , . . . , dn ). In the next section it is shown that the set M (d, v0 ) can be equipped with the structure of a g-crystal and that the bijection τn is a crystal isomorphism (if one replaces direct products of sets with tensor products of crystals and considers the set n S as a trivial crystal).

506

ANTON MALKIN

3. Levi restriction and the crystal structure on quiver varieties 3.1. A subquiver Q 0 Let Q 0 = (I 0 , H 0 ) ⊂ Q = (I, H ) be a (full) subquiver of Q (i.e., the set I 0 of vertices of Q 0 is a subset of the set I of vertices of Q, and two vertices of Q 0 are connected by an oriented edge h ∈ H 0 of Q 0 if and only if they are connected by an oriented edge of Q). 0 0 Let H Q Q (resp., H Q Q ) denote the set of edges h ∈ H of Q such that Out(h) ∈ Q 0 and In(h) ∈ / Q 0 (resp., In(h) ∈ Q 0 and Out(h) ∈ / Q 0 ). 0 In what follows, subscripts Q and Q are used to distinguish objects defined using the two quivers. For example, Q MsD,V is a quiver variety associated to Q (in particular, D and V are Z[I ]-graded C-linear spaces), whereas Q 0 MsD,V is a quiver variety associated to Q 0 (and, correspondingly, D and V are Z[I 0 ]-graded C-linear spaces or Z[I ]-graded spaces with zero components in degrees i ∈ I \I 0 ), and Q F is the path algebra of Q, whereas Q 0 F is the path algebra of Q 0 . Note that Q 0 F is a subalgebra of Q F . 3.2. The set Q Q 0 M (d, v0 , v) Let D and V be Z[I ]-graded C-linear spaces with graded dimensions d and v, respectively, and let D 0 , V 0 be Z[I 0 ]-graded C-linear spaces defined as follows:  M  Di0 = Di ⊕ VIn(h) , 0

h∈H Q Q Out(h)=i

Vi0 = Vi , for i ∈ I 0 ⊂ I . One has a regular map ζ Q Q 0 :

Q

3 D,V →

Q0

3 D 0 ,V 0 given by

ζ Q Q 0 ((x, p, q)) = (x 0 , p 0 , q 0 ), where x h0 = x h

for h ∈ H 0 ⊂ H, 0
h∈H Q Q pi0 = pi , {x h }In(h)=i , h∈H Q Q 0
 qi0 = qi , (ε(h))−1 x h Out(h)=i .

∗s Let Q Q 0 3∗s D,V be the set of all (x, p, q) ∈ Q 3 D,V such that ζ Q Q 0 ((x, p, q)) ∈ Q 0 3 D 0 ,V 0 . s,∗s s,∗s s ∗s Let Q Q 0 3 D,V = Q Q 0 3 D,V ∩ Q 3 D,V . Roughly speaking, Q Q 0 3 D,V is the open subset of 3 D,V consisting of all stable points that are also ∗-stable at vertices of Q 0 . s,∗s s,∗s s Let v0 ∈ Z[I ], and let Q Q 0 3s,∗s D,V,v0 = Q Q 0 3 D,V ∩ Q 3 D,V,v0 . Since Q Q 0 3 D,V,v0 is an s open G V -invariant subset of Q 3 D,V,v0 , one has the following proposition (cf. (2.5.4), (2.3.5)).

TENSOR PRODUCT VARIETIES AND CRYSTALS

PROPOSITION s,∗s Q Q 0 3 D,V,v0 is empty

507

or has pure dimension

dim Q Q 0 3s,∗s D,V,v0 =

1 1 hX v, vi + hX v0 , v0 i + hd, vi + hd, v0 i − hv0 , v0 i, 2 2

and G V -action on Q Q 0 3s,∗s D,V,v0 is free. It follows that Q Q 0 Ms,∗s (d, v0 , v) = variety of pure dimension dim Q Q 0 Ms,∗s (d, v0 , v) =

Q Q0

3s,∗s D,V,v0 /G V is naturally a quasi-projective

1 1 hX v, vi + hX v0 , v0 i 2 2 + hd, vi + hd, v0 i − hv0 , v0 i − hv, vi.

Let Q Q 0 M (d, v0 , v) be the set of irreducible components of Q Q 0 Ms,∗s (d, v0 , v). This set is also in a natural bijection (cf. the end of Sec. 2.5) with the sets of irres,∗s s,∗s s ducible components of Q Q 0 3s,∗s D,V,v0 and of Q Q 0 3 D,V,U = Q Q 0 3 D,V ∩ Q 3 D,V,U (where dim U = v − v0 ). 3.3. Levi restriction Given (x, p, q) ∈ Q 3sD,V , let Q Q 0 K (x, q) denote the maximal among the graded subspaces U ⊂ V , satisfying the following conditions: Ui = {0} for any i ∈ / I 0, xh U ⊂ U

for any h ∈ H 0 ,

xh U = 0

for any h ∈ H Q Q ,

qi U = 0

for any i ∈ I 0 .

0

(3.3.1)

Let w ∈ Z[I 0 ] ⊂ Z[I ], and let W be a graded subspace of V with dim W ∈ Z[I 0 ] ⊂ Z[I ]. Then Q Q 0 3sD,V,v0 ,w (resp., Q Q 0 3sD,V,v0 ,W ) denotes the set of all (x, p, q) ∈ Q 3sD,V,v0 such that dim Q Q 0 K (x, q) = w (resp., Q Q 0 K (x, q) = W ). V Note that Q Q 0 3sD,V,v0 ,w is a fibration over a graded Grassmannian Grw with fibers s isomorphic to Q Q 0 3 D,V,v0 ,W , where dim W = w. Let W be as above, and let T be a graded subspace of V complimentary to W . Then one has a regular map (cf. Sec. 2.5) νQ Q0 :

Q Q0

3sD,V,v0 ,W →

Q0

3W × Q Q 0 3s,∗s D,T,v0

given by
ν Q Q 0 ((x, p, q)) = x W W , (x T T , p T D , q DT ) .

508

ANTON MALKIN

The fiber of ν Q Q 0 over a point (x, (y, p, q)) ∈ Q 0 3W × Q Q 0 3s,∗s D,T,v0 is the same as the fiber of the map γ (cf. Sec. 2.5) for the quiver Q 0 over the point 0 0 0 (x, ζ Q Q 0 ((y, p, q))) ∈ Q 0 3W × Q 0 3s,∗s D 0 ,T 0 , where Ti = Ti for i ∈ I ⊂ I , and D and ζ Q Q 0 are defined in Section 3.2. Therefore an analogue of Proposition 2.5 holds, and one obtains the following bijection between sets of irreducible components: θQ Q0 : ∼

− →

Q

M (d, v0 , v) G
Q Q 0 M (d, v0 , v − u) × Q 0 M δ Q Q 0 (d, v), ρ Q Q 0 (v − u), ρ Q Q 0 (v) ,

u∈Z≥0 [I 0 ]

where
ρ Q Q 0 (v) i = vi for v ∈ Z[I ], i ∈ I 0 ⊂ I, X
δ Q Q 0 (d, v) i = di + vIn(h) for d, v ∈ Z[I ], i ∈ I 0 ⊂ I. 0

h∈H Q Q Out(h)=i

Union over v gives a bijection G ∼ θ Q Q 0 : Q M (d, v0 ) − →

Q Q0


M (d, v0 , x) × Q 0 M δ Q Q 0 (d, x), ρ Q Q 0 (x) .

x∈Z≥0 [I ]

The bijection θ Q Q 0 is an analogue of the restriction of a representation of g to a Levi subalgebra of a parabolic subalgebra of g. This construction is a straightforward generalization of the reduction to sl2 -subalgebras introduced by Lusztig [L1, Sec. 12] and used by Nakajima [N2] and Kashiwara and Saito [KS]. 3.4. Levi restriction and the second bijection for tensor product varieties Let D, V be Z[I ]-graded C-linear spaces, let D be an n-step partial flag in D, and let v be an n-tuple of elements of Z[I ]. Recall (cf. Sec. 2.6) that nQ 5sD,D,V,v is a locally closed subset of Q 3sD,V . Let n

Q Q0

5s,∗s = nQ 5sD,D,V,v ∩ Q Q 0 3s,∗s D,V . D,D,V,v

Then nQ Q 0 5s,∗s is an open subset of nQ 5sD,D,V,v , and therefore it is empty or has D,D,V,v pure dimension equal to that of nQ 5sD,D,V,v . Let nQ Q 0 T (d, d, v, v) be the set of irreducible components of nQ Q 0 5s,∗s . D,D,V,v The restriction of the map γ : Q 3 D,V,U → Q 3U × Q 3∗s D,T (cf. Sec. 2.5) to Q Q0

3s,∗s D,V,U =

s,∗s δ 3s,∗s D,V ∩ Q 3 D,V,U has the image equal to Q 3U × Q 3 D,T , where δ 0

Q Q0

is as in (2.5.2), and denotes the open subset of δQ 3U consisting of all x ∈ δQ 3U T h∈H ker x h = {0} for any i ∈ I 0 . It follows that the restriction of the such that Out(h)=i

δ 30 Q U

TENSOR PRODUCT VARIETIES AND CRYSTALS

509

map γ to nQ Q 0 5s,∗s ∩ 3 has the image equal to δQ 3U × nQ 5s,∗s , and one D,D,T,v D,D,V,v Q D,V,U can repeat the argument in (2.13) to get a bijection G ∼ n n 0 T (d, d, v, v) − → Q Q0 β n : Q S (d, d, v0 , v) × Q Q M (d, v0 , v). Q Q0 0

v0 ∈Z≥0 [I ]

3.5. Levi restriction and the first bijection for tensor product varieties Let v, v˜ be n-tuples of elements of Z[I ], let vˆ be an n-tuple of elements of Z[I 0 ] ⊂ P P P Z[I ] such that nk=1 vk + nk=1 v˜ k + nk=1 vˆ k = dim V , and let nQ Q 0 5s,∗s be D,D,V,v,˜v,ˆv s,∗s s n n a locally closed subset of Q Q 0 5 D,D,V,v ∩ Q 5 D,D,V,v,˜v+ˆv , consisting of all (x, p, q) ∈ n 5s,∗s ∩n 5s such that the dimension of the maximal graded subspace Q Q0 D,D,V,v Q D,D,V,v,˜v+ˆv ˆ k of p(Dk ), satisfying the conditions (cf. (3.3.1)) V
ˆ k ⊂ p(Dk−1 ) V for any i ∈ / I 0, i i ˆk ⊂ V ˆk xh V

for any h ∈ H 0 , 0

ˆ k ⊂ p(Dk−1 ) for any h ∈ H Q Q , xh V ˆ k ) ⊂ Dk−1 for any i ∈ I 0 , qi (V (3.5.1) i i Pk−1 l Pk−1 l Pk is equal to l=1 v + l=1 v˜ + l=1 vˆ l (cf. the definition of nQ 5sD,D,V,v,˜v in Sec. 2.6). ˆ k satisfying conditions (3.5.1) contains p(Dk−1 ), Note that the maximal subspace V Pk−1 l Pk−1 l Pk−1 l which has dimension l=1 v + l=1 v˜ + l=1 vˆ , and it is contained in q −1 (Dk−1 ), Pk−1 l Pk Pk which has dimension l=1 v + l=1 v˜ l + l=1 vˆ l . Let ˆ1 ⊂ V ˜ 1 ⊂ V1 ⊂ V ˆ2 ⊂ V ˜2 ⊂ ··· ⊂ V ˆn ⊂ V ˜ n ⊂ Vn = V ) V = (0 = V0 = V ˆ k − dim Vk−1 = vˆ k , be a 3n-step Z[I ]-graded partial flag in V such that dim V n k k k k k k ˜ ˆ ˜ dim V − dim V = v˜ , dim V − dim V = v , and let Q Q 0 5s,∗s be a locally D,D,V,V s,∗s s,∗s n n closed subset of Q Q 0 5 D,D,V,v consisting of all (x, p, q) ∈ Q Q 0 5 D,D,V,v such that ˜ k = q −1 (Dk−1 ) ∩ Vk , and V ˆ k is the maximal graded subspace of Vk Vk = p(Dk ), V satisfying conditions (3.5.1) (cf. the definition of nQ 5sD,D,V,V in Sec. 2.6). The variety nQ Q 0 5s,∗s is a fibration over the variety of 3n-step partial flags D,D,V,v,˜v,ˆv in V with dimensions of the subfactors given by v, v˜ , and vˆ , and a fiber of this fibration is isomorphic to nQ Q 0 5s,∗s . D,D,V,V One has a regular map (given by restrictions of x, p, and q) Q Q0

σn :

n

Q Q0

→

Q Q0

5s,∗s D,D,V,V

3s,∗s 1

ˆ 1 ,V ˜ 1 /V ˆ1 D /D0 ,V1 /V

× · · · × Q Q 0 3s,∗s n

× Q 0 3Vˆ 1 /V0 × · · · × Q 0 3Vˆ n /Vn−1 .

ˆ n ,V ˜ n /V ˆn D /Dn−1 ,Vn /V

510

ANTON MALKIN

Let D 0 , V 0 be Z[I 0 ]-graded C-linear spaces defined in Sec. 3.2, and let D0 = (0 = 0 0 0 ˜ 0 1 ⊂ V0 1 ⊂ V ˜ 02 ⊂ · · · ⊂ D ⊂ D 1 ⊂ · · · ⊂ D n = D 0 ) (resp., V0 = (0 = V 0 = V ˜ 0 n ⊂ V0 n = V 0 )) be an n-step partial flag in D 0 (resp., a 2n-step partial flag in V 0 ) V induced by the flags (0 = D0 ⊂ D1 ⊂ · · · ⊂ Dn = D) and (0 = V0 ⊂ V1 ⊂ · · · ⊂ ˆ 1 ⊂ V1 ⊂ V ˆ2 ⊂ ··· ⊂ V ˆ n ⊂ Vn = V )). Vn = V ) (resp., by the flag (0 = V0 = V Recall (cf. (2.12)) that 00

Q0

σn :

n

Q0

5s,∗s D 0 ,D0 ,V 0 ,V0

→

Q0

3s,∗s 01

0

0

0

˜ 1 D /D 0 ,V 1 /V

× · · · × Q 0 3s,∗s 0n

0

0

0

˜ n D /D n−1 ,V n /V

× Q 0 3V˜ 0 1 /V0 0 × · · · × Q 0 3V˜ 0 n /V0 n−1 is the map obtained by restricting x, p, and q. Let ζ Q0 Q 0 denote the restriction of the ⊂ Q 3 D,V . map ζ Q Q 0 (cf. Sec. 3.2) to nQ Q 0 5s,∗s D,D,V,V The following proposition can be proven by an inductive (in n) argument similar to the ones used in Sections 2.7 – 2.12. Since the proof is completely analogous to the proofs in these sections, it is omitted. PROPOSITION

One has the following. (3.5.2) The image of the map Q Q 0 σ n is equal to Q Q0

3s,∗s 1

ˆ 1 ,V ˜ 1 /V ˆ1 D /D0 ,V1 /V

× · · · × Q Q 0 3s,∗s n

ˆ n ,V ˜ n /V ˆn D /Dn−1 ,Vn /V

× X,

where X is the open subset of Q 0 3Vˆ 1 /V0 × · · · × Q 0 3Vˆ n /Vn−1 such that Q0

3s,∗s 01

0 0 ˜ 01 D /D 0 ,V 1 /V

× · · · × Q 0 3s,∗s 0n

0 0 ˜ 0n D /D n−1 ,V n /V

×X

is the image of the map Q 0 σ n ◦ ζ Q0 Q 0 . (3.5.3) The set nQ Q 0 5s,∗s is an open dense subset of the total space of a locally D,D,V,V trivial fibration over the image of Q Q 0 σ n with a smooth connected fiber and such that the restriction of the projection map onto nQ Q 0 5s,∗s is equal to D,D,V,V 0 σ . QQ n (3.5.4) The dimension of nQ Q 0 5s,∗s does not depend on v˜ and vˆ . (It depends D,D,V,v,˜v,ˆv only on d, d, v, and v.)

TENSOR PRODUCT VARIETIES AND CRYSTALS

Proposition 3.5, together with (2.12), implies that the map ing bijection of sets of irreducible components: Q Q0

αn : ∼

n

Q Q0

− →

T (d, d, v, v) G

n

Q0

511

Q Q0

σ n induces the follow-


S δ Q Q 0 (d, v), δ Q Q 0 (d, u), ρ Q Q 0 (v), ρ Q Q 0 (u)

u1 ,...,un ∈Z≥0 [I ] v−u1 −...−un ∈Z≥0 [I 0 ]

× Q Q 0 M (d1 , v1 , u1 ) × · · · × Q Q 0 M (dn , vn , un ),

(3.5.5)

where δ Q Q 0 (d, v) and ρ Q Q 0 (v) are as in Section 3.3, and (δ Q Q 0 (d, v))k = δ Q Q 0 (dk , vk ), (ρ Q Q 0 (v))k = ρ Q Q 0 (vk ). 3.6. Levi restriction and the tensor product decomposition Let Q Q 0 τ n be the bijection G
n S δ Q Q 0 (d, v), δ Q Q 0 (d, u), ρ Q Q 0 (v), ρ Q Q 0 (u) Q0 u1 ,...,un ∈Z≥0 [I ] v−u1 −...−un ∈Z≥0 [I 0 ]

× Q Q 0 M (d1 , v1 , u1 ) × · · · × Q Q 0 M (dn , vn , un ) G ∼ n 0 − → Q S (d, d, v0 , v) × Q Q M (d, v0 , v) v0 ∈Z≥0 [I ]

given by Q Q0

τn =

Q Q0

β n ◦ Q Q 0 α −1 n .

It follows from the definitions of the bijections Q Q 0 α n (cf. Sec. 3.5), Q Q 0 β n (cf. Sec. 3.4), and θ Q Q 0 (cf. Sec. 3.3) that the following diagram of bijections is commutative:

512

ANTON MALKIN

F u1 ,...,un ∈Z≥0 [I ] u1 +···+un =v

Q

M (d1 , v1 , u1 ) × · · · × Q M (dn , vn , un )

Qτn

θ Q Q 0 ×···×θ Q Q 0

 M (d1 , v1 , u1 − w1 ) × Q 0 M (δ Q Q 0 (d1 , u1 ), ρ Q Q 0 (u1 − w1 ), ρ Q Q 0 (u1 )) .. . n × Q Q 0 M (d , vn , un − wn ) × Q 0 M (δ Q Q 0 (dn , un ), ρ Q Q 0 (un − wn ), ρ Q Q 0 (un )) Q Q0

F u1 ,...,un ∈Z≥0 [I ] u1 +···+un =v w1 ,...,wn ∈Z≥0 [I 0 ]

(Id ×···×Id × Q 0 τ )◦P n

F x1 ,...,xn ∈Z≥0 [I ] v−x1 −...−xn ∈Z≥0 [I 0 ] w∈Z≥0 [I 0 ]

 M (d1 , v1 , x1 ) × · · · × Q Q 0 M (dn , vn , xn ) ×nQ 0 S (δ Q Q 0 (d, v), δ Q Q 0 (d, x), ρ Q Q 0 (v − w), ρ Q Q 0 (x)) × Q 0 M (δ Q Q 0 (d, v), ρ Q Q 0 (v − w), ρ Q Q 0 (v)) Q Q0

Q Q 0 τ n ×Id

F v0 ∈Z≥0 [I ] w∈Z≥0 [I 0 ]

n S (d, d, v , v) × 0 Q

× M (δ Q0

Q Q0



Q Q0

(d, v), ρ

M (d, v0 , v − w) (v − w), ρ Q Q 0 (v))

Q Q0

Id ×θ −1 0 QQ

F v0 ∈Z≥0 [I ]

n S (d, d, v , v)  × 0 Q

Q

M (d, v0 , v)o

(3.6.1) where P is the permutation that moves all even terms in the direct product to the right. Roughly speaking, the commutativity of diagram (3.6.1) means that the Levi restriction commutes with the tensor product decomposition. 3.7. Digression: The sl2 -case The Levi restriction bijections allow one to reduce the generic ADE case to the case of a quiver R with one vertex and no edges (which corresponds to sl2 ). This section

TENSOR PRODUCT VARIETIES AND CRYSTALS

513

contains a complete description of quiver varieties and tensor product varieties for the quiver R. Since R has only one vertex and no edges, the vertex and edge indices are omitted in the notation. Thus, given two (nongraded) C-linear spaces D and V with dimensions d and v, respectively, R 3 D,V is the variety of pairs ( p, q), where p ∈ HomC (D, V ), q ∈ HomC (V, D), and pq = 0. The open set R 3sD,V (resp., ∗s R 3 D,V ) consists of all ( p, q) ∈ R 3 D,V such that p is surjective (resp., q is injective). The map ( p, q) → qp ∈ EndC (D) provides an isomorphism between the quiver s,∗s variety R Ms,∗s D,V = R 3 D,V /G V (cf. Sec. 2.3) and the GL D -orbit in EndC (D) consisting of all t ∈ EndC (D) such that t 2 = 0 and rank t = v. Note that this orbit is empty (if 2v > d) or is a smooth connected quasi-projective variety of dimension 2v(d − v) (if 2v ≤ d), which confirms the corresponding statements in Section 2.3. Similarly, the map ( p, q) → (qp, ker p) provides an isomorphism between the quiver variety R Ms (d, v0 , v) = R 3sD,V,v0 /G V (cf. Sec. 2.4) and the variety of pairs (t, B), where t ∈ EndC (D), t 2 = 0, rank t = v0 , and B is a subspace of D such that im t ⊂ B ⊂ ker t, dim B = d − v. It follows that R Ms (d, v0 , v) is smooth connected if v0 ≤ v ≤ d − v0 and empty otherwise, and hence its set of irreducible components R M (d, v0 , v) is a one-element or empty set, respectively. Therefore R

M (d, v0 ) =

G R

v∈Z≥0

M (d, v0 , v) =

d−v G0 R

v=v0

M (d, v0 , v) =

d−v G0

{ R Ms (d, v0 , v)}.

v=v0

Endow this set with the following structure of an sl2 -crystal:
s R wt R M (d, v0 , v) = (d − 2v),
s R ε R M (d, v0 , v) = v − v0 ,
s R ϕ R M (d, v0 , v) = d − v − v0 , ( s
if v > v0 , R M (d, v0 , v − 1) s e ˜ M (d, v , v) = R R 0 0 if v ≤ v0 , ( s
if v < d − v0 , R M (d, v0 , v + 1) s ˜ f M (d, v , v) = R R 0 0 if v ≥ d − v0 .

(3.7.1)

Here the weight lattice of sl2 is identified with Z, and the indexes of ε, ϕ, e, ˜ and f˜ are omitted because sl2 has only one root. The set R M (d, v0 ) equipped with structure (3.7.1) is a highest-weight normal sl2 -crystal with highest weight (d − 2v0 ). In other words, it is isomorphic (as a crystal) to L ((d − 2v0 )) (the crystal of the canonical basis of the highest-weight irreducible representation with highest weight (d − 2v0 )). Let D 1 be a subspace of D, let D be the 2-step partial flag (0 ⊂ D 1 ⊂ D), let d be the pair (d1 , d2 ), where d1 = dim D 1 , d2 = d − dim D 1 , and let v = (v1 , v2 ) be

514

ANTON MALKIN

an element of Z ⊕ Z. The map ( p, q) → qp ∈ EndC (D) provides an isomorphism between the multiplicity variety 2R S D,D,V,v (cf. Sec. 2.6) and the variety of all t ∈ EndC (D) such that t 2 = 0, t D1 ⊂ D1, rank t = v, rank t| D 1 = v1 , rank t|(D/D 1 ) = v2 .

(3.7.2)

Similarly, the map ( p, q) → (qp, ker p) provides an isomorphism between the tensor product variety 2R T D,D,V,v (cf. Sec. 2.6) and the variety of pairs (t, B), where t ∈ EndC (D) satisfies conditions (3.7.2) and B is a subspace of D such that im t ⊂ B ⊂ ker t, dim B = d − v. Straightforward linear algebra considerations (cf. [Ma2]) show that the variety of all t ∈ EndC (D) satisfying conditions (3.7.2) is empty if v < v1 +v2 , v > d2 −v2 +v1 , or v > d1 − v1 + v2 , and is a smooth connected quasi-projective variety otherwise. Hence the right-hand side of the tensor decomposition bijection R τ2 (cf. Sec. 2.14) becomes G

2 R

S (d, d, v0 , v) × R M (d, v0 ) =

v0 ∈Z≥0

1 1 1 2 min(d2 −v2 +v G,d −v +v ) R

v0

M (d, v0 ).

=v1 +v2

The bijection

R

τ2 :

R

∼

M (d , v ) × R M (d , v ) − → 1

1

2

2

1 1 1 2 min(d2 −v2 +v G,d −v +v ) R

v0

M (d, v0 )

=v1 +v2

can be written as R


τ2 R M(d1 , v1 , u1 ), R M(d2 , v2 , u2 ) = R M(d, v0 , u1 + u2 ),

(3.7.3)

where v0 is described as follows. Fix a subspace B in D such that dim B = d −u1 −u2 and dim D 1 ∩B = d1 −u1 . Then v0 is the (unique) integer such that the set of operators t ∈ EndC (D) with rank t = v0 form an open subset in the set of all t satisfying the

TENSOR PRODUCT VARIETIES AND CRYSTALS

515

following conditions: t 2 = 0, t D1 ⊂ D1, rank t| D 1 = v1 , rank t|(D/D 1 ) = v2 , im t ⊂ B ⊂ ker t.

(3.7.4)

Again, elementary linear algebra (cf. [Ma2]) shows that v0 = min(u2 + v1 , d1 − u1 + v2 ).

(3.7.5)

Equality (3.7.5), together with the definition of the tensor product of crystals (cf. Sec. 1.3), implies the following theorem (cf. [Ma2]). THEOREM

The bijection R τ2 is a crystal isomorphism R

τ2 :

∼

R

M (d1 , v1 ) ⊗ R M (d2 , v2 ) − →

M

2 R

S (d, d, v0 , v) ⊗ R M (d, v0 ),

v0 ∈Z≥0

where d = (d1 , d2 ), v = (v1 , v2 ), d = d1 + d2 , and the set 2R S (d, d, v0 , v) is considered as a trivial sl2 -crystal. Remark. In [Ma2], definitions of tensor product and multiplicity varieties for sl2 are slightly different. Namely, the varieties in [Ma2] are fibrations over the Grassmannian of all subspaces D 1 in D with given dimension, and the fibers of these fibrations are the corresponding (tensor product or multiplicity) varieties as defined in this paper. The sets of irreducible components and the tensor decomposition bijection are the same here and in [Ma2]. 3.8. A crystal structure on Q M (d, v0 ) In this section it is shown that the set Q M (d, v0 ) (cf. Sec. 2.5) can be endowed with the structure of a g-crystal. This structure was introduced by Nakajima [N2] following an idea of Lusztig [L1, Sec. 12]. The first step is to define the weight function. Identify the weight lattice of g with Z[I ] (i.e., i ∈ I ⊂ Z[I ] is the ith fundamental weight). Then the weight function Q

wt :

Q

M (d, v0 ) → Z[I ]

is given by Q

wt(Z ) = d − 2v + Q X v

516

ANTON MALKIN

for Z ∈ Q M (d, v0 , v) ⊂ Q M (d, v0 ). Here Q X is the matrix defined in (2.1.1). The crucial property of the function Q wt is its good behavior with respect to the Levi restriction. Namely, ρ Q Q 0 (d − 2v + Q X v) = δ Q Q 0 (d, v) − 2ρ Q Q 0 (v) + Q 0 X ρ Q Q 0 (v), where δ Q Q 0 and ρ Q Q 0 are as in Section 3.3. This allows one to use the Levi restriction to sl2 -subalgebras to define the crystal structure on Q M (d, v0 ). More explicitly, let Ri be the subquiver of Q consisting of the vertex i and no edges. Recall (cf. Sec. 3.3) that there is a bijection G
∼ θ Q Ri : Q M (d, v0 ) − → Q Ri M (d, v0 , v) × Ri M di + ( Q X v)i , vi . (3.8.1) v∈Z≥0 [I ]

Consider the right-hand side of (3.8.1) an sl2 -crystal with the crystal structure coming from the second multiple (on which it is defined as in Sec. 3.7), and let εi =

Ri

ε ◦ θ Q Ri ,

ϕi =

Ri

ϕ ◦ θ Q Ri ,

Q Q

Q Q

e˜i = θ Q−1 ˜ ◦ θ Q Ri , Ri ◦ Ri e ˜ f˜i = θ Q−1 Ri ◦ Ri f ◦ θ Q Ri .

(3.8.2)

These formulas, together with the weight function Q wt, provide the structure of a g-crystal on the set Q M (d, v0 ). By abuse of notation, this crystal is also denoted by Q M (d, v0 ). PROPOSITION

One has the following. (3.8.3) If d − 2v0 + Q X v0 ∈ / Z≥0 [I ], then Q M (d, v0 ) is an empty set. (3.8.4) If d − 2v0 + Q X v0 ∈ Z≥0 [I ], then Q M (d, v0 ) is a highest-weight normal g-crystal with highest weight d − 2v0 + Q X v0 . (3.8.5) If d − 2v0 + Q X v0 = d 0 − 2v00 + Q X v00 , then the crystals Q M (d, v0 ) and 0 0 Q M (d , v0 ) are isomorphic. Proof Statement (3.8.3) follows from (2.3.4). Statement (3.8.4) follows directly from definitions and the fact that s Q M (d, v0 , v0 ) (a connected smooth variety) is the only element of Q M (d, v0 ) which is killed by Q e˜i for all i ∈ I . To prove this, let Z ∈ Q M (d, v0 ) be an irreducible component of Q 3sD,V,U , and let (x, p, q) be a generic point of Z . Let γ be as in Section 2.5, and let (x UU , (x T T , p T D , q DT )) = γ ((x, p, q)). If U 6= {0}, then there

TENSOR PRODUCT VARIETIES AND CRYSTALS

517

T UU 6= {0}(cf. [L1, Sec. 12]). It follows that exists i ∈ I such that h∈H ker x h Out(h)=i
T h∈H ker x h ∩ ker qi 6= {0}, and hence Q e˜i Z 6= 0. Therefore if Q e˜i Z = 0 for Out(h)=i

all i ∈ I , then U = {0}, which means Z = Q Ms (d, v0 , v0 ). To prove (3.8.5), note that as sets both Q M (d, v0 , v0 +u) and Q M (d 0 , v00 , v00 +u) are in natural bijections (induced by the vector bundles γ ; cf. Sec. 2.5) with the set of irreducible components of the variety δ 3U , where dim U = u, and δ = d − 2v0 + 0 0 0 Q X v0 = d −2v0 + Q X v0 . In this way one obtains a bijection between Q M (d, v0 , v0 + u) and Q M (d 0 , v00 , v00 + u), and it follows from definitions that this bijection is a crystal isomorphism. 3.9. The main theorem From now on, the quiver Q is fixed and thus omitted in the notation. The following is the main result of this paper. THEOREM

The bijection τn (cf. Sec. 2.14) is a crystal isomorphism M ∼ n τn : M (d1 , v1 ) ⊗ · · · ⊗ M (dn , vn ) − → S (d, d, v0 , v) ⊗ M (d, v0 ), v0 ∈Z≥0 [I ]

where the set n S (d, d, v0 , v) is considered as a trivial g-crystal. Note that both sides might be empty. Proof Because of the commutativity of diagram (2.14.4), it is enough to consider the case of n = 2, and because the definition of the crystal structure uses the Levi restriction that commutes with the tensor product decomposition (cf. (3.6.1)), it is enough to consider the case of g = sl2 . Now the theorem follows from Theorem 3.7. 3.10. Corollary Recall that L(µ) denotes the highest-weight irreducible representation of g with highest weight µ ∈ Z≥0 [I ], and recall that L (µ) denotes the crystal of the canonical basis of L(µ). The following proposition is a corollary of Theorem 3.9. PROPOSITION

One has the following. (3.10.1) The g-crystal M (d, v0 ) is isomorphic to L (µ), where µ = d − 2v0 + X v0 .

518

ANTON MALKIN

(3.10.2) The cardinal of the set n S (d, d, v0 , v) of irreducible components of the multiplicity variety n S(d, d, v0 , v) is equal to the multiplicity of L(µ) in L(µ1 ) ⊗ · · · ⊗ L(µn ), where µ = d − 2v0 + X v0 , µk = dk − 2vk + X vk . In other words,
|n S (d, d, v0 , v)| = dimC Homg L(µ), L(µ1 ) ⊗ · · · ⊗ L(µn ) . (3.10.3) The bijection αn (cf. Sec. 2.10) identifies the set n T (d, d, v, v) of irreducible components of the tensor product variety n T(d, d, v, v) with the weight subset of weight d − 2v + X v in the crystal L (µ1 ) ⊗ · · · ⊗ L (µn ), where µ1 , . . . , µn are as in (3.10.2). Proof The proposition follows from Theorems 3.9 and 1.5 and Propositions 3.8 and 2.12. Remark. Note that (3.10.2) is a generalization of a theorem due to Hall [H] (cf. [M, Chap. II]). Statement (3.10.1) was also proven by Saito using results of [KS]. 3.11. The extended Lie algebra g0 According to (3.8.5), the set of g-crystals {M (d, v0 )}d,v0 ∈Z≥0 [I ] contains isomorphic elements, which suggests that in the context of quiver varieties it is more natural to consider a central extension of g than g itself. Let g0 = g ⊕ t, where t is a Cartan subalgebra of g. Then g0 is a reductive Lie algebra. Identify the weight lattice of g0 with Z[I ] ⊕ Z[I ] in such a way that the projection Z[I ] ⊕ Z[I ] → Z[I ] given by (v, u) → v − u is the projection from the weight lattice of g0 onto the weight lattice of g. A weight (v, u) ∈ Z[I ] ⊕ Z[I ] is called integrable if u ∈ Z≥0 [I ] and v − u ∈ Z≥0 [I ]. Let Q + ⊂ Z≥0 [I ] ⊕ Z≥0 [I ] be the set of all integrable weights. A finite-dimensional representation of g0 is called integrable if the highest weights of all its irreducible components are integrable. Note that the category of finite-dimensional integrable representations is closed with respect to tensor products. One can endow the set M (d, v0 ) with the structure of a g0 -crystal as follows. The maps εi , ϕi , e˜i , and f˜i are given by (3.8.2). (Note that roots of g0 are roots of g.) The weight function is given by (cf. Sec. 3.8) wt0 (Z ) = (d − v + X v, v) ∈ Z[I ] ⊕ Z[I ]

(3.11.1)

for Z ∈ M (d, v0 , v) ⊂ M (d, v0 ). Note that this weight function commutes with the Levi restriction (cf. an analogous statement for the weight function wt in Sec. 3.8). The following theorem shows that nonempty elements of the set {M (d, v0 )}(d,v0 )∈Z≥0 [I ]⊕Z≥0 [I ]

TENSOR PRODUCT VARIETIES AND CRYSTALS

519

form a closed (with respect to tensor product) family of g0 -crystals which is isomorphic to the family of crystals of canonical bases of irreducible integrable representations of g0 . THEOREM

One has the following. (3.11.2) If (d − v0 + X v0 , v0 ) ∈ / Q + , then the set M (d, v0 ) is empty. (3.11.3) If (d − v0 + X v0 , v0 ) ∈ Q + , then the set M (d, v0 ), together with the weight function wt0 and the maps εi , ϕi , e˜i , and f˜i given by (3.8.2), is a highestweight normal g0 -crystal with highest weight (d − v0 + X v0 , v0 ). (3.11.4) The image of the weight function wt0 : M (d, v0 ) → Z[I ]⊕Z[I ] is contained in Z≥0 [I ] × Z≥0 [I ]. (3.11.5) The bijection τn (cf. Sec. 2.14) is an isomorphism of g0 -crystals M ∼ n τn : M (d1 , v1 )⊗· · ·⊗M (dn , vn ) − → S (d, d, v0 , v)⊗M (d, v0 ), v0 ∈Z≥0 [I ]

where the set n S (d, d, v0 , v) is considered a trivial g0 -crystal. (3.11.6) The family {M (d, v0 )} of g0 -crystals is isomorphic to the family {L (d, v0 )} of crystals of canonical bases of irreducible integrable representations of g0 . In (3.11.6), L (d, v0 ) denotes the crystal of the canonical basis of the irreducible integrable representation L(d, v0 ) with highest weight (d − v0 + X v0 , v0 ). Proof Proofs of (3.11.2), (3.11.3), (3.11.5), and (3.11.6) are analogous to the proofs of (3.8.3), (3.8.4), Theorem 3.9, and (3.10.1), respectively, or one can deduce the former statements from the latter. To prove (3.11.4), note that an element of the set M (d, v0 , v) is an irreducible component of the variety MsD,V,v0 (cf. Sec. 2.4), where dim D = d, dim V = v. A point of this variety is a stable triple (x, p, q). Since it is stable, P im pi + h∈H im x h = Vi . Therefore vi ≤ (d + X v)i , which implies (3.11.4). In(h)=i

Remark. It follows from (3.11.4) that all weights of a finite-dimensional integrable representation of g0 belong to Z≥0 [I ] × Z≥0 [I ].

520

ANTON MALKIN

A. Appendix. Another description of multiplicity varieties and the tensor product diagram A.1. Lusztig’s description of the variety Ms,∗s D,V Lusztig [L3], [L6], [L4] related varieties Ms,∗s D,V (for a fixed D and varying V ) to locally closed subsets in a certain variety Z D . This section contains an overview of Lusztig’s construction (see [L3], [L6], [L4] for more details). Recall (see Sec. 3.7) that in the sl2 -case the map ( p, q) → qp allows one to identify Ms,∗s D,V with a nilpotent orbit in EndC D (namely, with the orbit consisting of all operators t such that t 2 = 0, and rank t = dim V ). Lusztig’s construction is the direct generalization of this device. However, for a generic quiver one has to take care not only of the maps p and q but also of the map x (i.e., one has to include the path algebra F into the play). L Let C I = i∈I C be considered a semisimple C-algebra. Note that a Z[I ]-graded C-linear space is the same as a C I -module, and note that the path algebra F is a C I – C I algebra. Let F˜ be a C-algebra defined as follows. As a C-linear space, F˜ = L F ⊕ i∈I Cu i , and the multiplication ◦ in F˜ is given by X f ◦ f0 = f · θi · f 0 , i∈I

f ◦ u i = f · [i], u i ◦ f = [i] · f, u i ◦ u j = δi j u i , where f, f 0 ∈ F , i, j ∈ I , θi is as in (2.1.2), [i] ∈ F denotes the path of length zero starting and ending at a vertex i ∈ I , and · denotes the multiplication in the path algebra F . Multiplication by u i on the left and on the right endows F˜ with the structure of a C I –C I algebra. This algebra was introduced by Lusztig [L6, Sec. 2.1] (see also [L3, Sec. 2.4] and [L4, Sec. 2.1]). It is an associative algebra with unit (given P by i∈I u i ). Since the Dynkin graph is assumed to be of finite type, F˜ is finitely generated (cf. [L6, Lem. 2.2]). Let D be a C I -module, and let Z D be the set of all C I –C I algebra homomorphisms π : F˜ → EndC I (D) (in other words, the set of all representations of F˜ in D). The set Z D is naturally an affine variety. Let ϑ 0 be a regular map ϑ0 :

3s,∗s D,V → Z D

given by ϑ 0 ((x, p, q)) = π, where π([h 1 · · · h n ]) = qOut(h 1 ) x h 1 · · · x h n pIn(h n )

TENSOR PRODUCT VARIETIES AND CRYSTALS

521

for a path [h 1 · · · h n ] in F . Being constant on the orbits of G V , the map ϑ 0 induces a s,∗s s,∗s regular map ϑ : Ms,∗s D,V = 3 D,V /G V → Z D . Let Z D,V = ϑ(M D,V ) ⊂ Z D . One has 0 Z D,V = Z D,V 0 if dim V = dim V , and thus the notation Z D,v (where v = dim V ) is sometimes used instead of Z D,V . The following theorem is due to Lusztig. THEOREM

One has the following. (A.1.1) ([L3, Th. 5.5], [L4, Lem. 4.12c]) The map ϑ : Ms,∗s D,V → Z D,V is a homeomorphism both in Zariski and in the smooth topologies. (A.1.2) ([L4, Lem. 4.12c]) The set Z D,V is locally closed in Z D . F (A.1.3) ([L4, Lem. 4.12d]) One has Z D = v∈Z≥0 [I ] Z D,v . Theorem A.1 shows that varieties Ms,∗s D,V for various V can be glued together, which is crucial for a geometric construction of the tensor product. Nakajima has also considered a union of all Ms,∗s D,V for a fixed D (cf. [N2], [N3, Sec. 2.5]). He calls the resultant variety M0 (∞, d), where d = dim D. It follows from [L3, Th. 5.5] that M0 (∞, d) is homeomorphic to Z D . Given π ∈ Z D , one can find v ∈ Z≥0 [I ] such that π ∈ Z D,v as follows (cf. [L3, Sec. 2]). Let D˙ = F ⊗C I D. Then D˙ is naturally a left F -module. Let ˙ D) be given by $π ( f ⊗ d) = π( f )d, and let Kπ be the largest $π ∈ HomC I ( D, ˙ Kπ ) is finite F -submodule of D˙ contained in the kernel of $π . Then v = dim( D/ and π ∈ Z D,v . A.2. Multiplicity varieties Let D, V be C I -modules, and let D = ({0} = D0 ⊂ D1 ⊂ · · · ⊂ Dn = D) be an n-step C I -filtration of D. Let v ∈ (Z≥0 [I ])n , and let v = dim V . Recall (see Sec. 2.6) that the multiplicity variety S D,D,V,v is a subset of Ms,∗s D,V . Consider the restriction of the map ϑ : Ms,∗s → Z (see Sec. A.1) to S . It follows from the definition D D,D,V,v D,V of the multiplicity variety that this restriction provides a homeomorphism between the multiplicity variety S D,D,V,v and a subvariety Z D,D,v,v of Z D consisting of all π ∈ Z D such that • for any k = 1, . . . , n the C I -submodule Dk is an F˜ -submodule of D with respect to the representation π : F˜ → EndC I D, • π ∈ Z D,v , • π|Dk /Dk−1 ∈ Z Dk /Dk−1 ,vk ⊂ Z Dk /Dk−1 for any k = 1, . . . , n. This description of the multiplicity varieties is reminiscent of the definition of the Hall-Ringel algebra (cf. [R]) associated to the algebra F˜ . One should be careful, however, because the set Z D,v ⊂ Z D is, in general, a union of orbits of AutC I (D)

522

ANTON MALKIN

rather than a single orbit. A similar situation occurs in the geometric (quiver) construction of the positive part of a (quantum) universal enveloping algebra when the underlying quiver is not of finite type (cf. [L1], [S]). A.3. The tensor product diagram Let D be a C I -module, and let D = ({0} = D0 ⊂ D1 ⊂ · · · ⊂ Dn = D) be an n-step C I -filtration of D. By analogy with Lusztig’s restriction construction in the theory of canonical basis (cf. [L2]), one can consider the following diagram: p

i

Z D1 /D0 × · · · × Z Dn /Dn−1 ← − Z0 − → Z D.

(A.3.1)

Here Z 0 is the variety of all π ∈ Z D such that for any k = 1, . . . , n the C I -submodule Dk of D is an F˜ -submodule with respect to the representation π, the map i is the embedding, and the map p is the restriction. Note that the subset p −1 (Z D1 /D0 ,v1 × · · · × Z Dn /Dn−1 ,vn ) ∩ i −1 (Z D,V ) ⊂ Z 0

(A.3.2)

is homeomorphic to the multiplicity variety n S D,D,V,v . It follows that subset (A.3.2) has pure dimension and the number of its irreducible components is equal to
dimC Homg0 L(d, v), L(d1 , v1 ) ⊗ · · · ⊗ L(dn , vn ) , where d = dim D, dk = dim Dk − dim Dk−1 , v = dim V , and L(d, v) denotes the highest-weight irreducible representation of g0 with the highest weight (d − v + X v, v) (cf. Sec. 3.11). Tensor product varieties can also be described in the context of diagram (A.3.1). They are related to a resolution of singularities of Z 0 . F Let Z = Z D , where D ranges over (representatives of the) isomorphism I classes of C -modules. Diagram (A.3.1) can be used to equip the category of perF verse sheaves on Z constant with respect to the stratification Z D = Z D,V with a structure of a Tannakian category. The results of this paper concerning the crystal tensor product provide a step towards relating this category to the category of integrable finite-dimensional representations of the Lie algebra g0 . Acknowledgments. The author thanks Igor Frenkel for numerous discussions, Hiraku Nakajima for pointing out an error in a previous version of the paper, and the referee for valuable comments. References [AHDM] M. F. ATIYAH, N. J. HITCHIN, V. G. DRINFELD, and YU. I. MANIN, Construction of instantons, Phys. Lett. A 65 (1978), 185 – 187. MR 82g:81049 488

TENSOR PRODUCT VARIETIES AND CRYSTALS

[BM]

[BG] [CB] [G] [GL]

[H]

[J] [K1] [K2] [K3] [KS] [KL] [KP] [KN] [L1]

[L2] [L3] [L4] [L5]

523

W. BORHO and R. MACPHERSON, “Partial resolutions of nilpotent varieties” in Analysis

and Topology on Singular Spaces (Luminy, France, 1981), II, III, Ast´erisque 101/102, Soc. Math. France, Montrouge, 1983, 23 – 74. MR 85j:14087 482 A. BRAVERMAN and D. GAITSGORY, Crystals via the affine Grassmannian, Duke Math. J. 107 (2001), 561 – 575. MR 2002e:20083 482 W. CRAWLEY-BOEVEY, Geometry of the moment map for representations of quivers, Compositio Math. 126 (2001), 257 – 293. MR 2002g:16021 490 V. GINZBURG, Lagrangian construction of the enveloping algebra U (sln ), C. R. Acad. Sci. Paris S´er. I Math. 312 (1991), 907 – 912. MR 92c:17017 488 I. GROJNOWSKI and G. LUSZTIG, “On bases of irreducible representations of quantum GLn ” in Kazhdan-Lusztig Theory and Related Topics (Chicago, Ill., 1989), Contemp. Math. 139, Amer. Math. Soc., Providence, 1992, 167 – 174. MR 94a:20070 482 P. HALL, “The algebra of partitions” in Proceedings of the Fourth Canadian Mathematical Congress (Banff, 1957), Univ. of Toronto Press, Toronto, 1959, 147 – 159. MR 28:1074 481, 482, 518 A. JOSEPH, Quantum Groups and Their Primitive Ideals, Ergeb. Math. Grenzgeb. (3) 29, Springer, Berlin, 1995. MR 96d:17015 480, 483, 486 M. KASHIWARA, Crystalizing the q-analogue of universal enveloping algebras, Comm. Math. Phys. 133 (1990), 249 – 260. MR 92b:17018 483 , On crystal bases of the Q-analogue of universal enveloping algebras, Duke Math. J. 63 (1991), 465 – 516. MR 93b:17045 483, 485 , Crystal bases of modified quantized enveloping algebra, Duke Math. J. 73 (1994), 383 – 413. MR 95c:17024 483 M. KASHIWARA and Y. SAITO, Geometric construction of crystal bases, Duke Math. J. 89 (1997), 9 – 36. MR 99e:17025 479, 508, 518 D. KAZHDAN and G. LUSZTIG, Proof of the Deligne-Langlands conjecture for Hecke algebras, Invent. Math. 87 (1987), 153 – 215. MR 88d:11121 488 H. KRAFT and C. PROCESI, Closures of conjugacy classes of matrices are normal, Invent. Math. 53 (1979), 227 – 247. MR 80m:14037 481 P. B. KRONHEIMER and H. NAKAJIMA, Yang-Mills instantons on ALE gravitational instantons, Math. Ann. 288 (1990), 263 – 307. MR 92e:58038 488 G. LUSZTIG, Quivers, perverse sheaves, and quantized enveloping algebras, J. Amer. Math. Soc. 4 (1991), 365 – 421. MR 91m:17018 479, 485, 487, 488, 508, 515, 517, 522 , Introduction to Quantum Groups, Progr. Math. 110, Birkh¨auser, Boston, 1993. MR 94m:17016 522 , On quiver varieties, Adv. Math. 136 (1998), 141 – 182. MR 2000c:16016 481, 520, 521 , Quiver varieties and Weyl group actions, Ann. Inst. Fourier (Grenoble) 50 (2000), 461 – 489. MR 2001h:17031 520, 521 , Semicanonical bases arising from enveloping algebras, Adv. Math. 151 (2000), 129 – 139. MR 2001e:17033 481

524

[L6]

[M] [Ma1] [Ma2] [MV] [N1] [N2] [N3] [N4] [R] [S] [VV]

ANTON MALKIN

, “Constructible functions on varieties attached to quivers” in Studies in Memory of Issai Schur, Progr. Math. 210, Birkh¨auser, Boston, 2002. 481, 482, 490, 491, 501, 520 I. G. MACDONALD, Symmetric Functions and Hall Polynomials, 2d ed., Oxford Math. Monogr., Oxford Univ. Press, New York, 1995. MR 96h:05207 481, 482, 518 A. MALKIN, A generalization of Hall polynomials to ADE case, Internat. Math. Res. Notices 2001, 1195 – 1202. MR 2002k:17016 482 , Tensor product varieties and crystals: G L(N ) case, Trans. Amer. Math. Soc. 354 (2002), 675 – 704. MR 2002h:20065 482, 514, 515 I. MIRKOVIC´ and K. VILONEN, Perverse sheaves on affine Grassmannians and Langlands duality, Math. Res. Lett. 7 (2000), 13 – 24. MR 2001h:14020 482 H. NAKAJIMA, Instantons on ALE spaces, quiver varieties, and Kac-Moody algebras, Duke Math. J. 76 (1994), 365 – 416. MR 95i:53051 479, 481, 488, 489, 490 , Quiver varieties and Kac-Moody algebras, Duke Math. J. 91 (1998), 515 – 560. MR 99b:17033 479, 481, 482, 488, 489, 490, 492, 508, 515, 521 , Quiver varieties and finite-dimensional representations of quantum affine algebras, J. Amer. Math. Soc. 14 (2001), 145 – 238. MR 2002i:17023 488, 521 , Quiver varieties and tensor products, Invent. Math. 146 (2001), 399 – 449. CMP 1 865 400 482 C. M. RINGEL, “Hall algebras” in Topics in Algebra (Warsaw, 1988), Part 1, Banach Center Publ. 26, Part 1, PWN, Warsaw, 1990, 433 – 447. MR 93f:16027 481, 521 A. SCHOFIELD, Notes on constructing Lie algebras from finite-dimensional algebras, preprint, 1991. 522 M. VARAGNOLO and E. VASSEROT, Perverse sheaves and quantum Grothendieck rings, preprint, arXiv:math.QA/0103182 482

Department of Mathematics, Yale University, New Haven, Connecticut 06520-8283, USA; current: Department of Mathematics, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139-4307, USA; [email protected]

HEXAGONAL CIRCLE PATTERNS AND INTEGRABLE SYSTEMS: PATTERNS WITH CONSTANT ANGLES ALEXANDER I. BOBENKO and TIM HOFFMANN

Abstract Hexagonal circle patterns with constant intersection angles are introduced and studied. It is shown that they are described by discrete integrable systems of Toda type. Conformally symmetric patterns are classified. Circle pattern analogs of holomorphic mappings z c and log z are constructed as special isomonodromic solutions. Circle patterns studied in the paper include Schramm’s circle patterns with the combinatorics of the square grid as a special case. 1. Introduction In recent years the theory of circle packings and, more generally, of circle patterns has enjoyed a fast development and a growing interest among specialists in complex analysis and discrete mathematics. This interest was initiated by Thurston’s rediscovery of the Koebe-Andreev theorem (see [K]) about circle packing realizations of cell complexes of a prescribed combinatorics and by his idea about approximating the Riemann mapping by circle packings (see [T], [RS]). Since then many other remarkable facts about circle patterns have been established, such as the discrete maximum principle and Schwarz’s lemma (see [R]) and the discrete uniformization theorem (see [BS]). These and other results demonstrate a surprisingly close analogy to the classical theory and allow one to talk about an emergence of the “discrete analytic function theory” (see [DS]), containing the classical theory of analytic functions as a small circles limit. Approximation problems naturally lead to infinite circle patterns, for an analytic description of which it is advantageous to stick with fixed regular combinatorics. The most popular are hexagonal packings where each circle touches exactly six neighbors. The C ∞ -convergence of these packings to the Riemann mapping was established in [HS]. Another interesting and elaborated class with similar approximation properties to be mentioned here are circle patterns with the combinatorics of the square grid DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 3, Received 4 September 2001. Revision received 1 February 2002. 2000 Mathematics Subject Classification. Primary 52C26; Secondary 37K20, 37K60.

525

526

BOBENKO and HOFFMANN

introduced by O. Schramm [S]. The square grid combinatorics of Schramm’s patterns results in an analytic description that is closer to the Cauchy-Riemann equations of complex analysis than the one of the packings with hexagonal combinatorics. Various other regular combinatorics also have similar properties (see [H]). Although computer experiments give convincing evidence for the existence of circle packing analogs of many standard holomorphic functions (see [DS]), the only circle packings that have been described explicitly are Doyle spirals (which are analogs of the exponential function) (see [BDS]) and conformally symmetric packings (which are analogs of a quotient of Airy functions) (see [BH]). Schramm’s patterns are richer in explicit examples: discrete analogs of the functions exp(z), erf(z), Airy (see [S]) and z c , log(z) (see [AB]) are known. A natural question is, what property is responsible for this comparative richness of Schramm’s patterns? Is it due to the packing-pattern (or hexagonal-square) combinatorics difference? Or maybe it is the integrability of Schramm’s patterns which is crucial. Indeed, Schramm’s square grid circle patterns in conformal setting are known to be described by an integrable system (see [BP2]), whereas for the packings it is still unknown.∗ In the present paper we introduce and study hexagonal circle patterns with constant angles, which merge features of the two circle patterns discussed above. Our circle patterns have the combinatorics of the regular hexagonal lattice (i.e., of the packings) and the intersection properties of Schramm’s patterns. Moreover, the latter are included as a special case into our class. An example of a circle pattern studied in this paper is shown in Figure 1. Each elementary hexagon of the honeycomb lattice corresponds to a circle, and each common vertex of three hexagons corresponds to an intersection point of the corresponding circles. In particular, each circle carries six intersection points with six neighboring circles, and at each point there meet three circles. To each of the three types of edges of the regular hexagonal lattice (distinguished by their directions), we associate an angle 0 ≤ αn < π, n = 1, 2, 3, and require that the corresponding circles of the hexagonal circle pattern intersect at this angle. It is easy to see that αn ’s are subject to the constraint α1 + α2 + α3 = π. We show that despite the different combinatorics the properties and description of the hexagonal circle patterns with constant angles are quite parallel to those of Schramm’s circle patterns. In particular, the intersection points of the circles are described by a discrete equation of Toda type known to be integrable (see [A]). In Section 4 we present a conformal (i.e., invariant with respect to M¨obius transformations) description of the hexagonal circle patterns with constant angles and show that one can vary the angles αn , arbitrarily preserving the cross-ratios of the intersection points should be said that, generally, the subject of discrete integrable systems on lattices different from Zn is underdeveloped at present. The list of relevant publications has been almost exhausted by [A], [KN], and [ND]. ∗ It

HEXAGONAL CIRCLE PATTERNS

527

on circles; thus each circle pattern generates a two-parameter deformation family. Analytic reformulation of this fact provides us with a new integrable system possessing a Lax representation in (2 × 2)-matrices on the regular hexagonal lattice. Conformally symmetric hexagonal circle patterns are introduced in Section 5. These are defined as patterns with conformally symmetric flowers; that is, each circle with its six neighbors is invariant under a M¨obius involution (M¨obius 180◦ rotation). A similar class of circle packings has been investigated in [BH]. Let us also mention that a different subclass of hexagonal circle patterns—with the multiratio property instead of the angle condition—has been introduced and discussed in detail in [BHS]. In particular, it has been shown that this class is also described by an integrable system. Conformally symmetric circle patterns comprise the intersection set of the two known integrable classes of hexagonal circle patterns: “with constant angles” and “with multiratio property.” The corresponding equations are linearizable and can be easily solved. Further, in Section 6 we establish a rather remarkable fact. It turns out that Doyle circle packings and analogous hexagonal circle patterns with constant angles are built out of the circles with the same radii. Moreover, given such a pattern, one can arbitrarily vary the intersection angles αn , preserving the circle radii. Extending the intersection points of the circles by their centers, one embeds hexagonal circle patterns with constant angles into an integrable system on the dual Kagome lattice (see Sec. 7). Having included hexagonal circle patterns with constant angles into the framework of the theory of integrable systems, we get an opportunity to apply the immense machinery of the latter to study the properties of the former. This is illustrated in Section 8, where we introduce and study some isomonodromic solutions of our integrable systems on the dual Kagome lattice. The corresponding circle patterns are natural discrete versions of the analytic functions z c and log z. The results of Section 8 constitute an extension to the present, somewhat more intricate, situation of the similar construction for Schramm’s circle patterns with the combinatorics of the square grid (see [BP2], [AB]). 2. Hexagonal circle patterns with constant angles The present paper deals with hexagonal circle patterns, that is, circle patterns with the combinatorics of the regular hexagonal lattice (the honeycomb lattice). An example is shown in Figure 1. Each elementary hexagon of the honeycomb lattice corresponds to a circle, and each common vertex of three hexagons corresponds to an intersection point of the corresponding circles. In particular, each circle carries six intersection points with six neighboring circles, and at each intersection point exactly three circles meet. For an analytic description of hexagonal circle patterns, we introduce some con-

528

BOBENKO and HOFFMANN

Figure 1. Doyle hexagonal isotropic circle pattern

venient lattices and variables. First, we define the regular triangular lattice T L as the cell complex whose vertices are  V (T L ) = z = k + `ω + mω2 : k, `, m ∈ Z ,

where ω = exp

 2πi  , 3

whose edges are the nonordered pairs  E(T L ) = [z1 , z2 ] : z1 , z2 ∈ V (T L ), |z1 − z2 | = 1 ,

(1)

(2)

and whose 2-cells are all regular triangles with the vertices in V (T L ) and the edges in E(T L ). We use triples (k, `, m) ∈ Z3 as coordinates of the vertices. On the regular triangular lattice, two such triples are equivalent and should be identified if they differ by the vector (n, n, n) with n ∈ Z. The vertices of the regular triangular lattice correspond to centers and intersection points of hexagonal circle patterns. Associating one of the centers with the point k = ` = m = 0, we obtain the regular hexagonal sublattice H L with all 2-cells being the regular hexagons with the vertices in  V (H L ) = z = k + `ω + mω2 : k, `, m ∈ Z, k + ` + m 6≡ 0 (mod 3) (3) and the edges in  E(H L ) = [z1 , z2 ] : z1 , z2 ∈ V (H L ), |z1 − z2 | = 1 .

(4)

The cells and the vertices correspond to circles and to the intersection points of the hexagonal circle patterns, respectively. Natural labelling of the faces  F(H L ) = z = k + `ω + mω2 : k, `, m ∈ Z, k + ` + m ≡ 0 (mod 3) yields V (H L ) ∪ F(H L ) = V (T L ).

HEXAGONAL CIRCLE PATTERNS

529

Definition 2.1 ˆ defines a hexagonal circle pattern if the We say that a map w : V (H L ) 7→ C following condition is satisfied. • Let  πi  zk = z0 + εk ∈ V (H L ), k = 1, 2, . . . , 6, where ε = exp , 3 be the vertices of any elementary hexagon in H L with the center z0 ∈ ˆ lie on a circle, and F(H L ). Then the points w(z1 ), w(z2 ), . . . , w(z6 ) ∈ C their circular order is just the listed one. We denote the circle through the points w(z1 ), w(z2 ), . . . , w(z6 ) by C(z0 ), thus putting it into a correspondence with the center z0 of the elementary hexagon above. As a consequence of this condition, we see that if two elementary hexagons of H L with the centers in z0 , z00 ∈ F(H L ) have a common edge [z1 , z2 ] ∈ E(H L ), then the circles C(z0 ) and C(z00 ) intersect at the points w(z1 ) and w(z2 ). Similarly, if three elementary hexagons of H L with the centers in z0 , z00 , z000 ∈ F(H L ) meet in one point z0 ∈ V (H L ), then the circles C(z0 ), C(z00 ), and C(z000 ) also have a common intersection point w(z0 ). Remark. We also consider circle patterns defined not on the whole of H L but rather on some connected subgraph of the regular hexagonal lattice. To each pair of intersecting circles, we associate the corresponding edge e ∈ E(H L ) and denote by φ(e) the intersection angle of the circles, 0 ≤ φ < 2π. The edges of E(H L ) can be decomposed into three classes:  E 1H = e = [z0 , z00 ] ∈ E(H L ) : z0 − z00 = ±1 ,  E 2H = e = [z0 , z00 ] ∈ E(H L ) : z0 − z00 = ±ω ,  E 3H = e = [z0 , z00 ] ∈ E(H L ) : z0 − z00 = ±ω2 . (5) These three sets correspond to three possible directions of the edges of the regular hexagonal lattice. We study in this paper a subclass of hexagonal circle patterns intersecting at given angles defined globally on the whole lattice. Definition 2.2 ˆ defines a hexagonal circle pattern with conWe say that a map w : V (H L ) 7→ C stant angles if, in addition to the condition of Definition 2.1, the intersection angles of circles are constant within their class E nH , that is, if φ(e) = αn ,

∀e ∈ E nH , n = 1, 2, 3.

530

BOBENKO and HOFFMANN

This angle condition implies, in particular, α1 + α2 + α3 = π.

(6)

We call the circle pattern isotropic if all its intersection angles are equal, that is, if α1 = α2 = α3 = π/3. The existence of hexagonal circle patterns with constant angles can be easily demonstrated via solving a suitable Cauchy problem. For example, one can start with the initial circles C(n(1 − ω)), C(n(ω − ω2 )), C(n(ω2 − 1)), n ∈ N. Remark. In the special case of α1 = α2 = π/2, α3 = 0, one obtains two pairs of touching circles intersecting orthogonally at each vertex. The hexagonal circle pattern becomes in this case a circle pattern of Schramm [S] with the combinatorics of the square grid. The generalization α1 + α2 = π, α3 = 0 was introduced in [BP2]. Figure 2 shows a nearly Schramm pattern (α3 is small).

Figure 2. A nearly Schramm pattern

3. Point and radii descriptions In this paper three different analytic descriptions are used to investigate hexagonal circle patterns with constant angles. Obviously, these circle patterns can be characterized through the radii of the circles. On the other hand, they can be described through the coordinates of some natural points, such as the intersection points or the centers of circles. Finally, note that the class of circle patterns with constant angles is invariant

HEXAGONAL CIRCLE PATTERNS

531

ˆ with respect to arbitrary fractional-linear transformations of the Riemann sphere C (M¨obius transformations). Factorizing with respect to this group, we naturally come to a conformal description of the hexagonal circle patterns with constant angles in Section 4.

r1 r6 r5

α3

r2

α1 δ2

α δ3 2 δ4

δ1

α2

δ5

δ6 α1

α3

r3

r4

Figure 3. Circle flower

The basic unit of a hexagonal circle pattern is the flower, illustrated in Figure 3 and consisting of a center circle surrounded by six petals. The radius r of the central circle and the radii rn , n = 1, . . . , 6, of the petals satisfy arg

6 Y

(r + eiαn rn ) = π,

(7)

n=1

where αn is the angle between the circles with radii r and rn . Specifying this for the hexagonal circle patterns with constant angles and using (6), one obtains the following theorem. THEOREM 3.1 The mapping r : F(H L ) → R+ is the radius function of a hexagonal circle pattern with constant angles α1 , α2 , α3 if and only if it satisfies

arg

3 Y

(1 + eiαn Rn )(e−iαn + Rn+3 ) = 0,

n=1

for every flower.

Rn =

rn , r

(8)

532

BOBENKO and HOFFMANN

Conjugating (8) and dividing it by the product R1 · · · R6 , one observes that for every hexagonal circle pattern with constant angles there exists a dual one. Definition 3.2 Let r : F(H L ) → R+ be the radius function of a hexagonal circle pattern C P with constant angles. The hexagonal circle pattern C P ∗ with the same constant angles and the radii function r ∗ : F(H L ) → R+ given by r∗ =

1 r

(9)

is called dual ∗ to C P. For deriving the point description of the hexagonal circle patterns with constant angles, let us consider the intersection points jointly with the centers of the circles. Fix ˆ The reflections of this point in the circles of the pattern are some point P∞ ∈ C. called conformal centers of the circles. In the particular case of P∞ = ∞, the conformal centers become the centers of the corresponding circles. We call an extension of a circle pattern by conformal centers a center extension. M¨obius transformations play a crucial role for the considerations in this paper. Recall that the cross-ratio of four points q(z 1 , z 2 , z 3 , z 4 ) :=

(z 2 − z 1 )(z 4 − z 3 ) (z 3 − z 2 )(z 1 − z 4 )

(10)

is invariant with respect to these transformations. We start with a simple lemma. LEMMA 3.3 Let z 2 , z 4 be the intersection points, and let z 1 , z 3 be the conformal centers of two circles intersecting with the angle α, as in Figure 4. The cross-ratio of these points is

q(z 1 , z 2 , z 3 , z 4 ) = e−2iα .

(11)

The claim is obvious for the Euclidean centers. These can be mapped to conformal centers by an appropriate M¨obius transformation that preserves the cross-ratio. ˆ to (11) with the corThus a circle pattern provides a solution z : V (T L ) → C responding angles α. This solution is defined at the vertices of the lattice T L with quadrilateral sites composed of the pairs of points, as in Lemma 3.3. For a hexagonal circle pattern with constant angles, one can derive from (11) equations for the interˆ and conformal centers z : V (T L \ H L ) → C. ˆ section points z : V (H L ) → C ∗ Note

that in the theory of circle packings there is a notion of a dual packing which is completely different from the present definition.

HEXAGONAL CIRCLE PATTERNS

533

z4 α

z1

z3

z2 Figure 4. Cross-ratio of an elementary quadrilateral THEOREM 3.4 Let z, z 1 , z 2 , z 3 , z 4 , z 5 , z 6 be conformal centers of a flower of a hexagonal circle pattern with constant angles α1 , α2 , α3 , where αn , n = 1, 2, 3, are the angles of pairs of circles corresponding to z, z n and z, z n+3 . Define δ1 , δ2 , δ3 through

2αn = δn+2 − δn+1 (mod 2π),

n ∈ {1, 2, 3} (mod 3).

(12)

Then z n satisfy a discrete equation of Toda type on the hexagonal lattice 3 X n=1

An



 1 1 + = 0, z − zn z − z n+3

(13)

where An = eiδn+2 − eiδn+1 ,

n ∈ {1, 2, 3} (mod 3).

Let w1 , w2 , w3 be the intersection points neighboring the point w of a hexagonal circle pattern, and let αn be the angle between the circles intersecting at w, wn . Then the following identity holds: 3 X n=1

An



1  = 0. w − wn

(14)

In the special case of Schramm’s patterns α3 = 0, α1 + α2 = π, flowers contain only four petals, and we arrive at the following. THEOREM 3.5 The intersection points w, wr , wu , wl , wd ∈ C and the conformal centers z, zr , z u , zl , z d ∈ C of the neighboring circles of a Schramm pattern (labelled as in Fig. 5) satisfy the discrete equation of Toda type on a Z2 -lattice:

1 1 1 1 + = + , w − wr w − wl w − wu w − wd 1 1 1 1 + = + . z − zr z − zl z − zu z − zd

(15) (16)

534

BOBENKO and HOFFMANN wu zu wl zl

w z

wr zr

wd zd

Figure 5. Intersection points and conformal centers of a Schramm circle pattern

The proofs of these two theorems are presented in Appendix B (in the case of complex α ∈ C). It should be noticed that equations (16) and (13) both appeared in the theory of integrable equations in a totally different context (see [Su], [A]). The geometric interpretation in the present paper is new. The sublattices of the centers and of the intersection points are dependent, and one can be essentially uniquely reconstructed from the other. The corresponding formulas, which are natural to generalize for complex cross-ratios (10), hold for discrete equations of Toda type (13), (14) and (15), (16). We present these relations, which are of independent interest in the theory of discrete integrable systems, in Appendix B. In Section 7 we show that the hexagonal circle patterns with constant angles are described by an integrable system on a regular lattice closely related to the lattices introduced in Section 2. 4. Conformal description Let us now turn our attention to a conformal description of the hexagonal circle patterns. This description is used in the construction of conformally symmetric circle patterns in Section 5. We derive equations for the cross-ratios of the points of the hexagonal lattice which allow us to reconstruct the lattice up to M¨obius transformations. First, we investigate the relations of cross-ratios inside one hexagon of the hexagonal lattice shown in Figure 6. LEMMA 4.1 ˆ let z1 , . . . , z6 be six points of a hexagon cyclically Given a map w : V (H L ) → C, ordered (see Fig. 6). To each edge [zi , zi+1 ] of the hexagon, let us assign a cross-ratio Ti of successive points wi = w(zi ):

Ti := q(wi , wi+1 , wi+2 , wi−1 ) (i mod 6).

(17)

HEXAGONAL CIRCLE PATTERNS

535

w4

w3 T3

w5

T4

T2

T5

T1

T6

w6

w2

w1

Figure 6. Cross-ratios in a hexagon

Then the equations T3 T5 T1 + T3 − 1 − T1 T2 T3 T1 = = = T4 T6 T2 1 − T2

(18)

hold. Proof Let m i be the M¨obius transformation that maps wi−1 , wi , and wi+1 to 0, 1, and ∞, −1 respectively. Then Mi := m i+1 m i maps wi−1 , wi , and wi+1 to wi , wi+1 , and wi+2 , respectively, and has the form   1 −1 Mi = . 1 −Ti Now (M6 M5 M4 )−1 = ρ M3 M2 M1 for some ρ gives the desired identities. Since every edge in E(H L ) belongs to two hexagons, it carries two cross-ratios in general. We now investigate the relation between them and show that in the case of a hexagonal circle pattern with constant angles the two cross-ratios coincide. This way, Lemma 4.1 furnishes a map T : E(H L ) → C. Together with the cross-ratios of successive points in each hexagon (the Ti ), we need the cross-ratios of a point and its three neighbors. Definition 4.2 Let z1 , z2 , and z3 be the neighbors of z ∈ V (H L ) counterclockwise ordered, and let ˆ furnishes three cross-ratios [zi , z] ∈ E iH , i = 1, 2, 3. Any map w : V (H L ) → C in each point z ∈ V (H L ): Sz(i) := q(wi , wi+1 , w, wi+2 ),

i = 1, 2, 3 (mod 3).

(19)

They are linked by the modular transformation Sz(i+1) = 1 −

1 (i)

Sz

(i mod 3).

(20)

536

BOBENKO and HOFFMANN

Sz(i) 3 w4 w3 R w2 1

w1

w6 R2 w5 Sz(i) 2

Figure 7. Six points around one edge

Of course, the two types of cross-ratios are not independent. When we look at an edge from E(H L ) with its four neighboring edges, the six points form four cross-ratios (two T and two S). Three points fix the M¨obius transformation, and the other three can be calculated from only three cross-ratios. Therefore one expects one equation. LEMMA 4.3 ˆ and z1 , . . . , z6 as shown in Figure 7 and the cross-ratios Given w : V (H L ) → C

R1 := q(w1 , w2 , w4 , w3 ),

R2 := q(w5 , w2 , w6 , w3 ),

(21)

Sz(i) := q(w3 , w4 , w2 , w6 ), 3

Sz(i) := q(w2 , w5 , w3 , w1 ) 2

(22)

with [z2 , z3 ] ∈ E iH , the following identity holds: (i)

Sz3 1 − T2 R1 = = (i) . 1 − T1 R2 Sz2

(23)

Proof Use the fact that Ti and Ri are linked by the relation Ri :=

1 = q(wi , wi+1 , wi−1 , wi+2 ), 1 − Ti

(24)

and insert the definition of the cross-ratio. Everything written so far holds for any hexagonal lattice, but in the case of the hexagonal circle patterns with constant angles, we can calculate all S in terms of the αi only: sin αi+1 Sz(i) := q(wi , wi+1 , w, wi+2 ) = e−iαi (25) sin αi+2 if [zi , z] ∈ E i . This can be easily verified by applying a M¨obius transformation sending z to ∞. The three circles become straight lines forming a triangle with angles π − αi . The S (i) are now the quotients of two of its edges.

HEXAGONAL CIRCLE PATTERNS (i)

537 (i)

In particular, we get Sz1 = Sz2 along all edges [z1 , z2 ] ∈ E(H L ). Thus for our hexagonal circle patterns, formula (25) implies that there is only one T per edge since from equation (23) we get T1 = T2 . (26) ˆ So Lemma 4.1 defines, in fact, a map T : E(H L ) → C. (1) Note that we can get back the αi from, say, S in equation (25), by the following formulas: (1)

e

2iα1

S = (1) , S

e

2iα2

=

1 + S (1) 1+S

(1)

(1)

,

e

2iα3

S (1) 1 + S = (1) . 1 + S (1) S

(27)

Now we can formulate the main theorem, which states that the equations that link the cross-ratios in a hexagon plus constant S (1) describe for real-negative T a hexagonal circle pattern with constant angles up to M¨obius transformations. 4.4 ˆ for which Sz(1) = S for all z ∈ Given S ∈ C and a map w : V (H L ) → C ˆ V (H L ), the cross-ratios Ti of Lemma 4.1 define a map T : E(H L ) → C and obey equations (18). ˆ of (18), for each S ∈ C there Conversely, given a solution T : E(H L ) → C ˆ having is, up to M¨obius transformations, a unique map w : V (H L ) → C (1) the map T as cross-ratios in the sense of Lemma 4.1 and having Sz = S for all z ∈ V (H L ). If the solution T is real-negative, the resulting w defines a hexagonal circle pattern with constant angles.

THEOREM

(1)

(2)

Proof (1) This is proven above. (2) Starting with the points w2,0,0 , w1,1,0 , and w1,0,1 (which fixes the M¨obius (1) transformation), we can calculate w1,0,0 using S1,0,0 . Now we have three points for each hexagon touching in z1,0,0 . Using the T ’s, we can determine all their other points. Every new point has at least two determined neighbors, so we can use the S (1) to compute the third. Now we are able to compute all points of the new neighboring hexagons, and so on. Since real-negative cross-ratios imply that the four points lie cyclically ordered on a circle, we have a circle pattern in the case of real-negative map T . But with (25) the constancy of S (1) implies that the intersection angles are constant too. We have shown that hexagonal circle patterns with constant angles come in one complex or two real parameter families: one can choose S ∈ C, arbitrarily preserving T ’s.

538

BOBENKO and HOFFMANN

In Appendix A it is shown how this observation implies a Lax representation on the hexagonal lattice for the system (18). Figure 2 shows a nearly Schramm pattern. One can obtain Schramm’s description by taking combinations of Ti ’s and S (i) ’s which stay finite in the limit α3 → 0. 5. Conformally symmetric circle patterns The basic notion of conformal symmetry introduced in [BH] for circle packings can be easily generalized to circle patterns: every elementary flower is invariant under the M¨obius equivalent of a 180◦ rotation. Definition 5.1 (1) An elementary flower of a hexagonal circle pattern with petals Ci is called conformally symmetric if there exists a M¨obius involution sending Ci to Ci+3 (i mod 3). (2) A hexagonal circle pattern is called conformally symmetric if all of its elementary flowers are. For investigation of conformally symmetric patterns, we need the notion of the multiratio of six points: m(z 1 , z 2 , z 3 , z 4 , z 5 , z 6 ) :=

z1 − z2 z3 − z4 z5 − z6 · · . z2 − z3 z4 − z5 z6 − z1

(28)

Hexagonal circle patterns with multiratio −1 are discussed in [BHS] (see [KS] for a further geometric interpretation of this quantity). It turns out that in the case of conformally symmetric hexagonal circle patterns the two known integrable classes “with constant angles” and “with multiratio −1” coincide. 5.2 An elementary flower of a hexagonal circle pattern is conformally symmetric if and only if the opposite intersection angles are equal and the six intersection points with the central circle have multiratio −1. A hexagonal circle pattern is conformally symmetric if and only if it has constant intersection angles and for all circles the six intersection points have multiratio −1.

PROPOSITION

(1)

(2)

Proof First, let us show that six points z i have multiratio −1 if and only if there is a M¨obius involution sending z i to z i+3 . If there is such a M¨obius transformation, it is clear that q(z 1 , z 2 , z 3 , z 4 ) =

HEXAGONAL CIRCLE PATTERNS

539

q(z 4 , z 5 , z 6 , z 1 ) and m(z 1 , z 2 , z 3 , z 4 , z 5 , z 6 ) =

−q(z 1 , z 2 , z 3 , z 4 ) , q(z 4 , z 5 , z 6 , z 1 )

(29)

which implies m(z 1 , z 2 , z 3 , z 4 , z 5 , z 6 ) = −1.

(30)

Conversely, let M be the M¨obius transformation sending z 1 , z 2 , and z 3 to z 4 , z 5 , and z 6 , respectively. Then for z ∗ := M(z 4 ) equation (30) implies q(z 4 , z 5 , z 6 , z 1 ) = q(z 1 , z 2 , z 3 , z 4 ) = q(M(z 1 ), M(z 2 ), M(z 3 ), M(z 4 )) = q(z 4 , z 5 , z 6 , z ∗ ), and thus z∗ = z1 . The same computation yields z2 = M(z5 ) and z3 = M(z6 ). Now the first statement of the theorem is proven since the intersection points and angles determine the petals completely. For the proof of the second statement, the only thing left to show is that all flowers being conformally symmetric implies that the three intersection angles per flower sum up to π . So let us look at a flower around the circle C with petals Ci . Let the angle between C and Ci be αi (and we know that αi = αi+3 ). Then the angle β1 = π − α1 − α2 is the angle between C1 and C2 , and β3 = π − α3 − α1 is the one between C3 and C4 . Since the flowers around C2 and C3 are conformally symmetric too, we now have two ways to compute the angle β2 between C2 and C3 . Namely, π − α2 − α3 = β2 = π − (π − α3 − α1 ) − (π − α1 − α2 ), which implies α1 + α2 + α3 = π. Using (29), we see that in the case of multiratio −1 the opposite cross-ratios T defined in Section 4 must be equal: Ti = T(i mod 3) . Thus on E(H L ) the T ’s must be constant in the direction perpendicular to the edge they are associated with. For the three cross-ratios in a hexagon, we get from (18), T1 + T2 + T3 − T1 T2 T3 = 2.

(31)

Let us rewrite this equation by using the labelling shown in Figure 8. We denote by ak , b` , and cm the cross-ratios (24) associated with the edges of the families E 1H , E 2H , and E 3H , respectively. Note that for the labels in Figure 8, k+`+m =1 holds. Written in terms of ak , b` , cm , equation (31) is linear: ak + b` + cm = 1 and can be solved explicitly.

(32)

540

BOBENKO and HOFFMANN

a−n b−n a−n

cn

b−n an b−n

cn cn a−n

a−1 a1 c1 a0 b0 c0 c0 a0 b1

an an

bn bn bn c−n Figure 8. Cross-ratios for conformally symmetric circle patterns

LEMMA 5.3 The general solution to (32) on E(H L ) is given by

ak = a0 + k1, b` = b0 + `1, cm = c0 + m1,

(33)

with a0 , b0 , c0 ∈ C and 1 = 1 − a0 − b0 − c0 . Proof Obviously, (33) solves (32). On the other hand, it is easy to show that the cross-ratios on three neighboring edges determine all other cross-ratios recursively. Therefore (33) is the only solution to (32).

THEOREM 5.4 Conformally symmetric circle hexagonal circle patterns are described as follows: Given an , bn , cn by (33), choose S (1) ∈ C (or angles α1 and α2 ); then there is a conformally symmetric circle pattern with intersection angles α1 , α2 , and α3 given by formula (27).

HEXAGONAL CIRCLE PATTERNS

541

Figure 9. A conformally symmetric circle pattern and V (H L ) under the quotient √ of two Airy functions √ f (z) = (Bi(z) + 3Ai(z))/(Bi(z) − 3Ai(z))

The cross-ratio of four points can be viewed as a discretization of the Schwarzian derivative. In this sense conformally symmetric patterns correspond to maps with a linear Schwarzian. The latter are quotients of two Airy functions (see [BH]). Figure 9 shows that a symmetric solution of (32) is a good approximation of its smooth counterpart. Figure 2 also shows a conformally symmetric pattern with S (1) = 10i and a0 = b0 = 1/3 − 0.29, and c0 = 1 − 2a0 . 6. Doyle circle patterns Doyle circle packings are described through their radii function. The elementary flower of a hexagonal circle packing is a central circle with six touching petals (which, in turn, touch each other cyclically). Let the radius of the central circle be R, and let R1 , . . . , R6 be the radii of the petals. Doyle spirals are described through the constraint Rk Rk+3 = R 2 , Rk Rk+2 Rk+4 = R 3 . (34) There are two degrees of freedom for the whole packing—for example, R1 /R and R2 /R, which are constant for all flowers. The next lemma claims that the same circles that form a Doyle packing build up a circle pattern with constant angles.

542

BOBENKO and HOFFMANN

LEMMA 6.1 The radii function of a Doyle packing (i.e., a solution to (34)) solves the pattern radii equation (8) for any choice of the angles α1 , α2 , α3 = π − α1 − α2 .

Proof Insert identities (34) into (8). Definition 6.2 A hexagonal circle pattern with constant angles whose radii function obeys the constraint (34) is called a Doyle pattern. Figure 1 shows a Doyle pattern. The following lemma and theorem show how the Doyle patterns fit into our conformal description. LEMMA 6.3 Doyle patterns are conformally symmetric.

Proof We have to show that for a Doyle pattern the multiratio of each hexagon is −1. One can assume that the circumferencing circle has radius 1 and center zero and that w1 = 1. Then the other points are given by w j = w j−1

1 + R j eiα( j 1 + Rj

e−iα( j

mod 3) mod 3)

,

where R j are the radii of the petals. Inserting this into the definition of the multiratio implies the claim. THEOREM 6.4 Doyle patterns and their M¨obius transforms can be characterized in the following way: the corresponding solution to (32) is constant∗ ; that is, a0 + b0 + c0 = 1 and a0 , b0 , c0 > 0.

Proof Any Doyle pattern gives rise to a constant solution of (32) since all elementary flowers of a Doyle pattern are similar. On the other hand, one easily sees that all 0 < Ri < 1 can be realized. ∗ Here

constant means ak = a0 , bk = b0 , ck = c0 for all k ∈ Z.

HEXAGONAL CIRCLE PATTERNS

543

7. Lax representation and dual patterns We start with a general construction of integrable systems on graphs which does not hang on the specific features of the lattice. This notion includes the following ingredients: • an oriented graph G with the vertices V (G ) and the edges E(G ); • a loop group G[λ], whose elements are functions from C into some group G (the complex argument λ of these functions is known in the theory of integrable systems as the spectral parameter); • a wave function 9 : V (G ) 7→ G[λ] defined on the vertices of G ; • a collection of transition matrices L : E(G ) 7→ G[λ] defined on the edges of G. It is supposed that for any oriented edge e = (zout , zin ) ∈ E(G ) the values of the wave functions in its ends are connected via 9(zin , λ) = L(e, λ)9(zout , λ).

(35)

Therefore the following zero-curvature condition has to be satisfied. Consider any closed contour consisting of a finite number of edges of G : e1 = (z1 , z2 ),

e2 = (z2 , z3 ),

...,

e p = (z p , z1 ).

Then L(e p , λ) · · · L(e2 , λ)L(e1 , λ) = I. In particular, for any edge e = (z1 , z2 ) one has e−1 = (z2 , z1 ) and
−1 L(e−1 , λ) = L(e, λ) .

(36)

(37)

Actually, in applications the matrices L(e, λ) also depend on a point of some set X (the phase space of an integrable system), so that some elements x(e) ∈ X are attached to the edges e of G . In this case, the discrete zero-curvature condition (36) becomes equivalent to the collection of equations relating the fields x(e1 ), . . . , x(e p ) attached to the edges of each closed contour. We say that this collection of equations admits a zero-curvature representation. Such representation may be used to apply analytic methods for finding concrete solutions, transformations, or conserved quantities. In this paper we deal with zero-curvature representations on the hexagonal lattice H L and especially on a lattice closely related to it: a special quadrilateral lattice QL , which is obtained from H L by deleting from E(T L ) the edges of the hexagonal lattice E(H L ) (i.e., corresponding to the intersection points). The so-defined lattice QL has quadrilateral cells, vertices V (QL ) = V (T L ), and the edges E(QL ) = E(T L ) \ E(H L ),

544

BOBENKO and HOFFMANN

l

QH (2,1,−3)

k

m

HH Figure 10. Quadrilateral lattice QL

as shown in Figure 10. The lattice dual to QL is known as the Kagome lattice (see [B]). As in (5), there are three types of edges in E(QL ) distinguished by their directions:  Q E 1 = e = [z0 , z00 ] ∈ E(QL ) : z0 − z00 = ±1 ,  Q E 2 = e = [z0 , z00 ] ∈ E(QL ) : z0 − z00 = ±ω ,  Q E 3 = e = [z0 , z00 ] ∈ E(QL ) : z0 − z00 = ±ω2 . (38) There is a natural labelling (k, `, m) ∈ Z3 of the vertices V (QL ) which respects the lattice structure of QL . Let us decompose V (QL ) = V0 ∪ V1 ∪ V−1 into three sublattices:  V0 = z = k + `ω + mω2 : k, `, m ∈ Z, k + ` + m = 0 ,  V1 = z = k + `ω + mω2 : k, `, m ∈ Z, k + ` + m = 1 ,  V−1 = z = k + `ω + mω2 : k, `, m ∈ Z, k + ` + m = −1 . (39) Note that this definition associates a unique triple (k, `, m) to each vertex. Neighboring vertices of QL , that is, those connected by edges, are characterized by the property that their (k, `, m)-labels differ only in one component. Each vertex of V0 has six edges, whereas the vertices of V±1 have only three.

HEXAGONAL CIRCLE PATTERNS

545

The (k, `, m)-labelling of the vertices of QL suggests that we consider this lattice to be the intersection of Z3 with the strip |k + ` + m| ≤ 1. This description turns out to be useful, especially for the construction of discrete analogs of z c and log z in Section 8. We denote p = (k, `, m) ∈ Z3 and three different types of edges of Z3 by

E 1 = p, p + (1, 0, 0) , E 2 = p, p + (0, 1, 0) ,
E 3 = p, p + (0, 0, 1) , p ∈ Z3 . The group G[λ] which we use in our construction is the twisted loop group over SL(2, C):  (40) G[λ] = L : C 7→ SL(2, C) L(−λ) = σ3 L(λ)σ3 , σ3 = diag(1, −1). ∗ To each type E n , we associate a constant 1n ∈ C. Let z k,`,m , z k,`,m be fields defined ∗ 3 at vertices z, z : Z → C and satisfying ∗ ∗ ) = 1n − z out (z in − z out )(z in

(41)

on all the edges e = ( pout , pin ) ∈ E n , n = 1, 2, 3. Here we denote z out,in = ∗ z( pout,in ), z out,in = z ∗ ( pout,in ). We call the mapping z ∗ dual to z. To each oriented edge e = ( pout , pin ) ∈ E n , we attach the following element of the group G[λ]:   1 λ(z in − z out ) (n) 2 −1/2 . (42) L (λ) = (1 − λ 1n ) ∗ − z∗ ) λ(z in 1 out Note that this form, as well as condition (41), is independent of the orientation of the edge. Substituting (41) into (42), one obtains L(λ) in terms of the field z only. Dual fields z, z ∗ can be characterized in their own terms. The condition that for ∗ , z ∗ ) defined by (41) sum any quadrilateral of the dual lattice the oriented edges (z out in up to zero implies the following statements. THEOREM 7.1 (i) Let z, z ∗ : Z3 → C be dual fields, that is, fields satisfying duality condition (41). Then three types of elementary quadrilaterals of Z3 have the following cross-ratios:

11 , 13 13 q(z k,`,m , z k,`,m+1 , z k,`−1,m+1 , z k,`−1,m ) = , 12 12 q(z k,`,m , z k,`+1,m , z k−1,`+1,m , z k−1,`,m ) = . 11 q(z k,`,m , z k+1,`,m , z k+1,`,m−1 , z k,`,m−1 ) =

(43)

The same identities hold with z replaced by z ∗ . (ii) Given a solution z : Z3 → C to (43), formula (41) determines (uniquely up to translation) a solution z ∗ : Z3 → C of the same system (43).

546

BOBENKO and HOFFMANN

It is obvious that the zero-curvature condition (36) is fulfilled for every closed contour in Z3 if and only if it holds for all elementary quadrilaterals. It is easy to check that the transition matrices (42) satisfy the zero-curvature condition. THEOREM 7.2 Let z, z ∗ : Z3 → C be a solution to (41), and let e1 , e2 , e3 , e4 be consecutive positively oriented edges of an elementary quadrilateral of QL . Then the zero-curvature condition L(e4 , λ)L(e3 , λ)L(e2 , λ)L(e1 , λ) = I

holds with L(e, λ) defined by (42). Moreover, let the elements   1 λf (n) 2 −1/2 L (e, λ) = (1 − λ 1n ) , f g = 1n , λg 1

n = 1, 2, 3,

of G[λ] be attached to oriented edges e = ( pout , pin ) ∈ E n . Then the zero-curvature condition on Z3 is equivalent to the existence of z, z ∗ : Z3 → C such that the factorization f (e) = z( pin ) − z( pout ), g(e) = z ∗ ( pin ) − z ∗ ( pout ) holds. So-defined z, z ∗ satisfy (41) and (43). The zero-curvature condition in Theorem 7.2 is a generalization of the Lax pair found in [NC] for the discrete conformal mappings (see [BP1]). The zero-curvature condition implies the existence of the wave function 9 : Z3 → G[λ]. The last one can be used to restore the fields z and z ∗ . There holds the following result having many analogs in the differential geometry described by integrable systems (“Sym formula”; see, e.g., [BP2]). THEOREM 7.3 Let 9( p, λ) be the solution of (35) with the initial condition 9( p = 0, λ) = I . Then the fields z, z ∗ may be found as ! d9k,`,m 0 z k,`,m − z 0,0,0 = . (44) ∗ ∗ z k,`,m − z 0,0,0 0 dλ λ=0

This simple observation turns out to be useful for analytic constructions of solutions, in particular, in Section 8. Interpreting the lattice QL as {(k, `, m) ∈ Z3 : |k + ` + m| ≤ 1}, one arrives at the following.

HEXAGONAL CIRCLE PATTERNS

547

COROLLARY 7.4 Theorems 7.1, 7.2, and 7.3 hold for the lattice QL if in the statements one replaces Q Z3 by QL , p by z, E n by E n and assumes that k + ` + m = 0 in (43).

Let us return to hexagonal circle patterns and explain their relation to the discrete integrable systems of this section. To obtain the lattice QL , consider the intersection points of a hexagonal circle pattern jointly with the conformal centers of the circles. ˆ consists of the intersection The image of so-defined mapping z : V (QL ) → C points z(V1 ∪ V−1 ) and the conformal centers z(V0 ). The edges connecting the points on the circles with their centers correspond to the edges of the quadrilateral lattice QL . Whereas the angles αn are associated to three types E nH , n = 1, 2, 3, of the edges of the hexagonal lattice, constants δn defined in (12) are associated to three Q types E n , n = 1, 2, 3, of the edges of the quadrilateral lattice. Identifying them with 1n = e−iδn

(45)

in the matrices (42), one obtains a zero-curvature representation for hexagonal circle patterns with constant angles. THEOREM 7.5 ˆ of a The intersection points and the conformal centers of the circles z : QL → C hexagonal circle pattern with constant angles αn , n = 1, 2, 3, satisfy the cross-ratio system (43) on the lattice QL with 1n , n = 1, 2, 3, determined by (12), (45).

Theorem 7.5 follows from the identification of (43) with (11) using (12) and (45). The duality transformation (41) for arbitrary mappings satisfying the cross-ratio equations preserves the class of such mappings coming from circle patterns with constant angles. The last one is nothing but the dual circle pattern of Definition 3.2. 7.6 ˆ be a hexagonal circle pattern C P with constant angles together Let z : QL → C with the Euclidean centers of the circles. Then the dual circle pattern C P ∗ , together ˆ with the Euclidean centers of the circles, is given by the dual mapping z ∗ : QL → C. THEOREM

Since 1n , n = 1, 2, 3, are unitary (see (45)), relation (9) follows directly from (41). The patterns have the same intersection angles since the cross-ratio equations (43) for z and z ∗ coincide.

548

BOBENKO and HOFFMANN

8. The z c and log z patterns To construct hexagonal circle pattern analogs of holomorphic functions z c and log z, we use the analytic point description presented in Section 7. Recall that we interpret the lattice QL of the intersection points and the centers of the circles as a subset |k +`+m| ≤ 1 of Z3 and thus study the zero-curvature representation and 9-function on Z3 . A fundamental role in the presentation of this section is played by a nonautonomous constraint for the solutions of the cross-ratio system (43), (z k+1,`,m − z k,`,m )(z k,`,m − z k−1,`,m ) z k+1,`,m − z k−1,`,m (z k,`+1,m − z k,`,m )(z k,`,m − z k,`−1,m ) + 2(` − a2 ) z k,`+1,m − z k,`−1,m (z k,`,m+1 − z k,`,m )(z k,`,m − z k,`,m−1 ) + 2(m − a3 ) , (46) z k,`,m+1 − z k,`,m−1

2 bz k,`,m + cz k,`,m + d = 2(k − a1 )

where b, c, d, a1 , a2 , a3 ∈ C are arbitrary. Note that the form of the constraint is invariant with respect to M¨obius transformations. Our presentation in this section consists of three parts. First, we explain the origin of the constraint (46), deriving it in the context of isomonodromic solutions of integrable systems. Then we show that it is compatible with the cross-ratio system (43); that is, there exist nontrivial solutions of (43), (46). And finally, we specify parameters of these solutions to obtain circle pattern analogs of holomorphic mappings z c and log z. We choose a different gauge of the transition matrices to simplify formulas. Let us orient the edges of Z3 in the direction of increasing k + ` + m. We conjugate L (n) (λ) of positively oriented edges with the matrix diag(1, λ), and then we multiply by (1 − 1n λ2 )1/2 in order to get rid of the normalization of the determinant. Then writing µ for λ2 , we end up with the matrices   1 z in − z out , 1n L (n) (e, µ) =  (47) µ 1 z in − z out associated to the edge e = ( pout , pin ) ∈ E n oriented in the direction of increasing k +`+m. Each elementary quadrilateral of Z3 has two consecutive positively oriented pairs of edges e1 , e2 and e3 , e4 . The zero-curvature condition turns into L (n 1 ) (e2 )L (n 2 ) (e1 ) = L (n 2 ) (e4 )L (n 1 ) (e3 ).

Then the values of the wave function 8 in neighboring vertices are related by the

HEXAGONAL CIRCLE PATTERNS

formulas

549

 (1)   8k+1,`,m (µ) = L (e, µ)8k,`,m (µ), e ∈ E 1 , 8k,`+1,m (µ) = L (2) (e, µ)8k,`,m (µ), e ∈ E 2 ,   8 (3) (e, µ)8 k,`,m (µ), e ∈ E 3 . k,`,m+1 (µ) = L

(48)

We call a solution z : Z3 7→ C of equations (43) isomonodromic (cf. [I]) if there exists a wave function 8 : Z3 7→ GL(2, C)[µ] satisfying (48) and some linear differential equation in µ: d 8k,`,m (µ) = Ak,`,m (µ)8k,`,m (µ), (49) dµ where Ak,`,m (µ) are (2 × 2)-matrices, meromorphic in µ, with poles whose position and order do not depend on k, `, m. It turns out that the simplest nontrivial isomonodromic solutions satisfy the constraint (46). Indeed, since det L (n) (µ) vanishes at µ = 1/1n , the logarithmic derivative of 8(µ) must be singular in these points. We assume that these singularities are as simple as possible, that is, simple poles. THEOREM 8.1 Let z : Z3 → C

be an isomonodromic solution to (43) with the matrix Ak,`,m in (49)

of the form

(n)

3 Ck,`,m X Bk,`,m + Ak,`,m (µ) = µ µ − 1/1n

(50)

n=1

(n)

with µ-independent matrices Ck,`,m , Bk,`,m and normalized ∗ trace tr A0,0,0 (µ) = 0. ∗ As

explained in the proof of Theorem 8.1, this normalization can be achieved without loss of generality.

550

BOBENKO and HOFFMANN

Then these matrices have the following form: Ck,`,m (1)

1 = 2

2 −bz k,`,m − c/2 bz k,`,m + cz k,`,m + d b bz k,`,m + c/2

k − a1 z k+1,`,m − z k−1,`,m  z k+1,`,m − z k,`,m · 1 ` − a2 = z k,`+1,m − z k,`−1,m  z k,`+1,m − z k,`,m · 1 m − a3 = z k,`,m+1 − z k,`,m−1  z k,`,m+1 − z k,`,m · 1

! ,

Bk,`,m =

(2)

Bk,`,m

(3)

Bk,`,m

(z k+1,`,m − z k,`,m )(z k,`,m − z k−1,`,m ) z k,`,m − z k−1,`,m



(z k,`+1,m − z k,`,m )(z k,`,m − z k,`−1,m ) z k,`,m − z k,`−1,m



(z k,`,m+1 − z k,`,m )(z k,`,m − z k,`,m−1 ) z k,`,m − z k,`,m−1



+

a1 I, 2

+

a2 I, 2

+

a3 I, 2

and z k,`,m satisfies (46). Conversely, any solution z : Z3 → C to the system (43), (46) is isomonodromic with Ak,`,m (µ) given by the formulas above. The proofs of Theorems 8.1 and 8.2 are presented in Appendix 9. We prove Theorem 8.1 by computing the compatibility conditions of (48) and (49), and we prove Theorem 8.2 by showing the solvability of a reasonably posed Cauchy problem. 8.2 For arbitrary b, c, d, a1 , a2 , a3 ∈ C, the constraint (46) is compatible with the crossratio equations (43); that is, there are nontrivial solutions of (43), (46). THEOREM

Further, we deal with the special case of (46) where b = a1 = a2 = a3 = 0. If c 6= 0, one can always assume that d = 0 in (46) by shifting z → z − (z k+1,`,m − z k,`,m )(z k,`,m − z k−1,`,m ) z k+1,`,m − z k−1,`,m (z k,`+1,m − z k,`,m )(z k,`,m − z k,`−1,m ) + 2` z k,`+1,m − z k,`−1,m (z k,`,m+1 − z k,`,m )(z k,`,m − z k,`,m−1 ) . + 2m z k,`,m+1 − z k,`,m−1

d : c

cz k,`,m = 2k

(51)

HEXAGONAL CIRCLE PATTERNS

551

To define a circle pattern analog of z c , it is natural to restrict the mapping to the following two subsets of Z3 :  Q = (k, `, m) ∈ Z3 k ≥ 0, ` ≥ 0, m ≤ 0 ,  H = (k, `, m) ∈ Z3 m ≤ 0 ⊂ Z3 . COROLLARY 8.3 The solution z : Q → C of the system (43) satisfying the constraint (51) is uniquely determined by its values z 1,0,0 , z 0,1,0 , z 0,0,−1 . (52)

Proof Using the constraint (51), one determines the values along the coordinate lines z n,0,0 , z 0,n,0 , z 0,0,−n , ∀n ∈ N. Then all other z k,`,m , k, `, −m ∈ N, in consecutive order are determined through the cross-ratios (43). Computations using different crossratios give the same result due to the following lemma about the eighth point. LEMMA 8.4 Let H ( p1 + 1/2, p2 + 1/2, p3 − 1/2), p = ( p1 , p2 , p3 ) ∈ Z3 , be the elementary hexahedron (lying in C) with the vertices z p+(k,`,−m) , k, `, m ∈ {0, 1}, and let the cross-ratios of the opposite faces of H ( p1 + 1/2, p2 + 1/2, p3 − 1/2) be equal to

q(z p , z p+(0,0,−1) ,z p+(0,1,−1) , z p+(0,1,0) ) = q(z p+(1,0,0) , z p+(1,0,−1) , z p+(1,1,−1) , z p+(1,1,0) ) =: q1 , q(z p , z p+(1,0,0) ,z p+(1,0,−1) , z p+(0,0,−1) ) = q(z p+(0,1,0) , z p+(1,1,0) , z p+(1,1,−1) , z p+(0,1,−1) ) =: q2 , q(z p , z p+(0,1,0) ,z p+(1,1,0) , z p+(1,0,0) ) = q(z p+(0,0,−1) , z p+(0,1,−1) , z p+(1,1,−1) , z p+(1,0,−1) ) =: q3 , with q1 q2 q3 = 1. Then all the vertices of the hexahedron are uniquely determined through four given points, z p , z p+(1,0,0) , z p+(0,1,0) , z p+(0,0,−1) . The relation of solutions of (43), (51) to circle patterns is established in the following. THEOREM 8.5 The solution z : Q → C of the system (43), (51) with the initial data

z 1,0,0 = 1,

z 0,1,0 = eiβ ,

z 0,0,−1 = eiγ

(53)

and unitary cross-ratios qn = e−2iαn determines a circle pattern. For all (k, `, m) ∈

552

BOBENKO and HOFFMANN

Q with even k + ` + m, the points z k±1,`,m , z k,`±1,m , z k,`,m±1 lie on a circle with the center z k,`,m . Proof We say that a quadrilateral is of the kite form if it has two pairs of equal adjacent edges (and that a hexahedron is of the kite form if all its six faces are of the kite form). These quadrilaterals have unitary cross-ratios with the argument equal to the angle between the edges (see Lem. 3.3). Our proof of the theorem is based on two simple observations: • if a quadrilateral has a pair of adjacent edges of equal length and unitary crossratio, then it is of the kite form; • if the cross-ratio of a quadrilateral is equal to q(z 1 , z 2 , z 3 , z 4 ) = e−2iα , where α is the angle between the edges (z 1 , z 2 ) and (z 2 , z 3 ) (as in Fig. 4), then the quadrilateral is of the kite form. Applying these observations to the elementary hexahedron in Lemma 8.4, one obtains the following. LEMMA 8.6 Let three adjacent edges of the elementary hexahedron in Lemma 8.4 have equal length, and let the cross-ratios of the face be unitary qn = e−2iαn . Then the hexahedron is of the kite form.

The constraint (51) with ` = m = 0 implies by induction |z 2n+1,0,0 − z 2n,0,0 | = |z 2n,0,0 − z 2n−1,0,0 |, and all the points z n,0,0 lie on the real axis. The same equidistance and straight-line properties hold true for the ` and m axes. Also, by induction, one shows that all elementary hexahedra of the mapping z : Q → C are of the kite form. Indeed, due to Lemma 8.6, the initial conditions (53) imply that the hexahedron H (1/2, 1/2, −1/2) is of kite form. Since the axis vertices lie on the straight lines, we get, for example, that the angle between the edges (z 1,1,0 , z 1,0,0 ) and (z 1,0,0 , z 2,0,0 ) is α3 . Now applying the observation (ii) to the faces of H (3/2, 1/2, −1/2), we see that |z 2,0,0 − z 1,0,0 | = |z 2,0,0 − z 2,1,0 | = |z 2,0,0 − z 2,0,−1 |. Lemma 8.6 implies that H (3/2, 1/2, −1/2) is of the kite form. Proceeding further this way and controlling the lengths of the edges meeting at z k,`,m with even k +`+m and the angles at z k,`,m with odd k +`+m, one proves the statement for all hexahedra.

HEXAGONAL CIRCLE PATTERNS

553

Denote the vertices of the hexagonal grid by (see Fig. 10)  Q H = (k, `, m) ∈ Q : |k + ` + m| ≤ 1 ,  H H = (k, `, m) ∈ H : |k + ` + m| ≤ 1 . The half-plane H H consists of the sector Q H and its two images under the rotations with the angles ±π/3. 8.7 The mapping z : H H → C given by (43), (51) with unitary cross-ratios qn = e−2iαn and unitary initial data COROLLARY

|z ±1,0,0 | = |z 0,±1,0 | = |z 0,0,0 | = 1 is a hexagonal circle pattern with constant angles αn together with the Euclidean centers of the circles. The centers of the corresponding circles are the images of the points with k + ` + m = 0. The circle patterns determined by most of the initial data β, γ ∈ R are quite irregular. But for a special choice of these parameters one obtains a regular circle pattern that we call the hexagonal circle pattern z c motivated by the asymptotic of the constraint (51) as k, `, m → ∞. Definition 8.8 The hexagonal circle pattern z c , 0 < c < 2 (together with the Euclidean centers of the circles), is the solution z : H H → C of (43), (51) with 1n+1 = e2iαn , 1n+2

n (mod 3),

and the initial conditions z 1,0,0 = 1,

z 0,1,0 = eic(α1 +α2 ) ,

z −1,0,0 = eicπ ,

z 0,0,−1 = eicα2 ,

z 0,−1,0 = e−icα3 .

(54)

Motivated by our computer experiments (see, in particular, Fig. 11) and the corresponding result for Schramm’s circle patterns with the combinatorics of the square grid (see [AB], [Ag]), we conjecture that the hexagonal z c is embedded; that is, the interiors of all elementary quadrilaterals (z(z1 ), z(z2 ), z(z3 ), z(z4 )), |zi+1 − zi | = 1, i (mod 4), are disjoint. In the isotropic case when all the intersection angles are the same, α1 = α2 = α3 =

π , 3

554

BOBENKO and HOFFMANN

Figure 11. Nonisotropic and isotropic circle patterns z 2/3

one has 11 = ω12 = ω2 13 ,

(55)

and due to the symmetry, it is enough to restrict the mapping to Q H . The initial conditions (54) become z 1,0,0 = 1,

z 0,1,0 = e2πic/3 ,

z 0,0,−1 = eπic/3 .

(56)

As for the smooth z c , the images of the coordinate axes arg z = π/6, π/3, π/2 are the axes with the arguments πc/6, πc/3, and π c/2, respectively: arg z n,0,−n = arg z n,n,m =

πc , 3

πc , 6

arg z 0,n,−n =

πc , 2

m = −2n; −2n ± 1, ∀n ∈ N.

For c = 6/q, q ∈ N, the circle pattern z c (Q H ), together with its additional q − 1 copies rotated by the angles 2πn/q, n = 1, . . . , q − 1, comprises a circle pattern covering the whole complex plane. This pattern is hexagonal at all circles other than the central one, which has q neighboring circles. The circle pattern z 6/5 is shown in Figure 12. Definition 8.8 is given for 0 < c < 2. For c = 2, the radii of the circles intersecting the central circle of the pattern become infinite. To get finite radii, the central circle should degenerate to a point z 0,0,0 = z 1,0,0 = z 0,0,−1 = z 0,1,0 = 0.

(57)

HEXAGONAL CIRCLE PATTERNS

555

Figure 12. Circle pattern z 6/5

The mapping z : Q → C is then uniquely determined by z 2,0,0 , z 0,2,0 , z 0,0,−2 , z 1,1,0 , z 1,0,−1 , z 0,1,−1 , the values of which can be derived taking the limit c → 2−0. Indeed, consider the quadrant m = 0 with the cross-ratios of all elementary quadrilaterals equal to q = e−2iα . Choosing z 1,0,0 = , z 0,1,0 = eicα as above, we get z 2,0,0 = 2/(2 − c). Normalization c = 2 − 2 yields z 2,0,0 = 1,

z 0,2,0 = eicα .

(58)

Since the angles of the triangle with the vertices z 0,0,0 , z 1,0,0 , z 1,1,0 are cα/2, π − α, α, respectively, one obtains z 1,1,0 =  + R eiα ,

R = 

sin(cα/2) . sin(α)

In the limit  → 0 we have

sin α iα e . (59) α Observe that in our previous notation α = α1 + α2 = π − α3 . Finally, the same computations for the sectors {` = 0, k ≥ 0, m ≤ 0} and {k = 0, ` ≥ 0, m ≤ 0} provide us with the following data: z 1,1,0 =

z 0,0,−2 = e2iα2 , z 0,2,0 = e2i(α1 +α2 ) , sin α2 iα2 sin α1 i(α1 +2α2 ) e , z 0,1,−1 = e , = α2 α1 sin(α1 + α2 ) i(α1 +α2 ) z 1,1,0 = e . α1 + α2

z 2,0,0 = 1, z 1,0,−1

(60)

In the same way, the initial data for other two sectors of H H are specified. The hexagonal circle pattern with these initial data and c = 2 is an analog of the holomorphic mapping z 2 . The duality transformation preserves the class of circle patterns we defined.

556

THEOREM

BOBENKO and HOFFMANN

8.9

We have ∗

(z c )∗ = z c ,

c∗ = 2 − c,

where (z c )∗ is the hexagonal circle pattern dual to the circle pattern z c and normalized to vanish at the origin (z c )∗ (z = 0) = 0. Proof Let us consider the mapping on the whole Q. It is easy to see that on the axes the duality transformation (41) preserves the form of the constraint, with c being replaced by c∗ = 2 − c. Then the constraint with c∗ holds for all points of Q due to the compatibility in Theorem 8.2. Restriction to |k + ` + m| ≤ 1 implies the claim. The smooth limit of the duality transformation of the hexagonal patterns is the following transformation f 7→ f ∗ of holomorphic functions:
0 f ∗ (z) =

1 f 0 (z)

.

The dual of f (z) = z 2 is, up to a constant, f ∗ (z) = log z. Motivated by this observation, we define the hexagonal circle pattern log z as the dual to the circle pattern z2: log z := (z 2 )∗ . The corresponding constraint (see (46)) 2k

(z k+1,`,m − z k,`,m )(z k,`,m − z k−1,`,m ) z k+1,`,m − z k−1,`,m (z k,`+1,m − z k,`,m )(z k,`,m − z k,`−1,m ) + 2` z k,`+1,m − z k,`−1,m (z k,`,m+1 − z k,`,m )(z k,`,m − z k,`,m−1 ) + 2m =1 z k,`,m+1 − z k,`,m−1

(61)

can be derived as a limit c → +0 (see [AB] for this limit in the square grid case). The initial data for log z are dual to the ones for z 2 . In our model case of the quadrant m = 0 factorizing q = 11 /12 with 11 = 1/2, 12 = e2iα /2, one arrives at the following data dual to (57), (58), and (59): z 0,0,0 = ∞, z 0,1,0 = iα,

z 1,0,0 = 0,

z 0,2,0 =

1 + iα, 2

1 , 2 α = eiα . 2 sin α

z 2,0,0 = z 1,1,0

(62)

In the isotropic case, the circle patterns are more symmetric and can be described as mappings of Q H .

HEXAGONAL CIRCLE PATTERNS

557

Definition 8.10 The isotropic hexagonal circle patterns z 2 and log z are the mappings Q H → C with the cross-ratios of all elementary quadrilaterals equal∗ to e−2πi/3 . The mapping z 2 is determined by the constraint (51) with c = 2 and the initial data z 0,0,0 = ∞,

z 1,0,−1

z 1,0,0 = 0,

z 2,0,0 = 1, √ 3 3 πi/3 = e , 2π

z 0,0,−1 = 0,

z 0,1,0 = 0,

z 0,0,−2 = e2πi/3 , z 0,2,0 = e4πi/3 , √ √ 3 3 3 3 2πi/3 z 0,1,−1 = − , z 1,1,0 = e . 2π 4π

For log z the corresponding constraint is (61) and the initial data are z 0,0,0 = ∞, z 2,0,0 =

z 1,0,0 = 0,

z 0,0,−1 =

π i, 3

z 0,1,0 =

2π i, 3

1 π 1 2π z 0,0,−2 = + i, z 0,2,0 = + i, 2 3 2 3     π 1 π 1 = z 0,1,−1 = √ +i , √ + 3i , 6 6 3 3  π 1 z 1,1,0 = − √ +i . 3 3

1 , 2

z 1,0,−1

The isotropic hexagonal circle patterns z 2 and log z are shown in Figure 13.

Figure 13. Isotropic circle patterns z 2 and log z

Starting with z c , c ∈ (0, 2], one can easily define z c for arbitrary c by applying some simple transformations of hexagonal circle patterns. The construction here is the same as for Schramm’s patterns (see [AB, Sec. 6] for details). Applying the inversion of the complex plane z 7→ 1/z to the circle pattern z c , c ∈ (0, 2], one obtains a circle ∗ The

first point in the cross-ratio is a circle center, and the quadrilaterals are positively oriented.

558

BOBENKO and HOFFMANN

pattern, satisfying the constraint with −c. It is natural to call it the hexagonal circle pattern z −c , c ∈ (0, 2]. Constructing the dual circle pattern, we arrive at a natural definition of z 2+c . Intertwining the inversion and the dualization, one constructs circle patterns z c for any c. In particular, inverting and then dualizing z = k + `ω + mω2 with 11 = −3, 12 = −3ω2 , 13 = −3ω, we obtain the circle pattern corresponding to z 3 : z k,`,m = (k + `ω + mω2 )3 − (k + ` + m). Note that this is the central extension corresponding to P∞ = 0. The points with even k + ` + m can be replaced by the Euclidean centers of the circles. As is shown in [AB, Sec. 6], the replacement of P∞ = 0 by P∞ = ∞ preserves the constraint (51). 9. Concluding remarks We restricted ourselves to the analysis of geometric and algebraic properties of hexagonal circle patterns, leaving the approximation problem beyond the scope of this paper. The convergence of circle patterns with the combinatorics of the square grid to the Riemann mapping has been proven by Schramm in [S]. We expect that his result can be extended to the hexagonal circle patterns defined in this paper. The entire circle pattern erf(z) also found in [S] remains rather mysterious. We were unable to find its analog in the hexagonal case, and thus we have no counterexamples to the Doyle conjecture for hexagonal circle patterns with constant angles. It seems that Schramm’s erf(z) is a feature of the square grid combinatorics. The construction of the hexagonal circle pattern analogs of z c and log z in Section 8 was based on the extension of the corresponding integrable system to the lattice Z3 . Theorem 8.5 claims that in this way one obtains a circle pattern labelled by three independent indices k, `, m. Fixing one of the indices, say, m = m 0 , one obtains a Schramm circle pattern with the combinatorics of the square grid. In the same way, the restriction of this three-dimensional pattern to the sublattice |k + ` + m − n 0 | ≤ 1 with some fixed n 0 ∈ Z yields a hexagonal circle pattern. In Section 8 we have defined circle patterns z c on the sublattices m 0 = 0 and n 0 = 0. In the same way, the sublattices m 0 6= 0 and n 0 6= 0 can be interpreted as the circle pattern analogs of the analytic function (z + a)c , a 6= 0, with the square and hexagonal grid combinatorics, respectively. It is unknown whether the theory of integrable systems can be applied to hexagonal circle packings. As already mentioned in the introduction, the underdevelopment of the theory of integrable systems on the lattices different from Zn may be a reason for this. We hope that integrable hexagonal circle patterns introduced in this paper and in [BHS] will lead to progress in hexagonal circle packings.

HEXAGONAL CIRCLE PATTERNS

559

Appendices Appendix A. A Lax representation for the conformal description We now give a Lax representation for equations (18). For z ∈ V (H L ), let [zi , z] ∈ E iH , and let m z be the M¨obius transformation that sends (z1 , z2 , z3 ) to (0, 1, ∞). For (1) and T . They are e = [z, z˜] ∈ E iH , set L i = m z˜ ◦ m −1 e z . The L i depend only on S given in equation (63). Note that we do not need to orient the edges since L i L i−1 is the identity if we normalize det L i = 1. The claim is that the closing condition, when multiplying the L i around one hexagon, is equivalent to equations (18). THEOREM A.1 Attach to each edge of E i a matrix L i (T, S) of the form  S−1  −1 S , L 1 (T, S) = T +S−1 1 S−1

TS

L 2 (T, S) = L 3 (T, S) =

(S−1) S 1+TS−1



1−S −S

1−TS −T S

! ,

−1 + T ( 1S − 1) + S S−1

 ;

(63)

then the zero-curvature condition for each hexagon in F(H L ), L 1 (T4 , S)L 2 (T5 , S)L 3 (T6 , S) = ρ L 3 (T3 , S)L 2 (T2 , S)L 1 (T1 , S)

(64)

for all S, is equivalent to (18). Proof The theorem is proved by straightforward calculations. Equations (63) and (64) are a Lax representation for equations (18) with the spectral parameter S. Appendix B. Discrete equations of Toda type and cross-ratios In the following we briefly discuss the connection between discrete equations of Toda type and cross-ratio equations for the square grid and the dual Kagome lattice. This interrelationship holds in a more general setting, namely, for discrete Toda systems on graphs. This situation will be considered in a subsequent publication. We start with the square grid. Let us decompose the lattice Z2 into two sublattices  Z2k = (m, n) ∈ Z2 (m + n mod 2) = k , k = 0, 1.

560

BOBENKO and HOFFMANN

THEOREM B.1 ˆ be a solution to the cross-ratio equation Let z : Z2 → C

q(z m,n , z m+1,n , z m+1,n+1 , z m,n+1 ) = q.

(65)

Then, restricted to the sublattices Z20 or Z21 , it satisfies the discrete equation of Toda type (16), z m,n

1 1 1 1 + = + . (66) − z m+1,n+1 z m,n − z m−1,n−1 z m,n − z m+1,n−1 z m,n − z m−1,n+1

ˆ of equation (66) and arbitrary q, w ∈ C, ˆ Conversely, given a solution z : Z20 → C 2 ˆ with z 1,0 = w there exists a unique extension of z to the whole lattice z : Z → C satisfying the cross-ratio condition (65). Moreover, restricted to the other sublattice ˆ satisfies the same discrete equation of Toda Z21 , the so-defined mapping z : Z21 → C type (66). Proof For any four complex numbers, the cross-ratio equation can be written in a simple fraction form: q(u 1 , u 2 , u 3 , u 4 ) = q ⇔

1 q q −1 − + = 0. u3 − u2 u3 − u4 u3 − u1

(67)

To distinguish between the two sublattices, we denote the points belonging to Z20 by z i , while wi are the points associated with Z21 . Now (67) gives four equations for a

z4 w3 z3

w4

z w2

z1

z1 w1 z2

w6

z6 w5 z5

q1 z

w4

w1

q2 q3 w3

z4 Figure 14. Nine points of Z2 and 13 points of V (T L )

z2 w2 z3

HEXAGONAL CIRCLE PATTERNS

561

point z and its eight neighbors, as shown in Figure 14: 1 z − z1 1 (q −1 − 1) z − z2 1 (q − 1) z − z3 1 (q −1 − 1) z − z4 (q − 1)

1 1 − , z − w1 z − w4 1 1 = q −1 − , z − w2 z − w1 1 1 =q − , z − w3 z − w2 1 1 = q −1 − . z − w4 z − w3 =q

(68)

Multiplying the second and fourth equation by q and taking the sum over all four equations, (68) yields equation (66). Conversely, given w1 , the first three equations of (68) determine w2 , w3 , and w4 uniquely. The so-determined w3 and w4 satisfy the fourth equation of (68) (for any choice of w1 ) if and only if (66) holds. This proves the second part of the theorem. Obviously, interchanging w and z implies the claim for the other sublattice. Now we pass to the dual Kagome lattice shown in Figure 10, where a similar relation holds for the discrete equation of Toda type on the hexagonal lattice. However, the symmetry between the sublattices is lost in this case. Here we decompose V (T L ) into V (H L ) and V (T L ) \ V (H L ) ∼ = F(H L ). B.2 ˆ of the Given q1 , q2 , q3 ∈ C with q1 q2 q3 = 1 and a solution z : V (T L ) → C cross-ratio equations THEOREM

q(z k,`,m−1 , z k,`,m , z k,`+1,m , z k,`+1,m−1 ) = q1 , q(z k+1,`,m , z k,`,m , z k,`,m−1 , z k+1,`,m−1 ) = q2 , q(z k,`−1,m , z k,`,m , z k+1,`,m , z k+1,`−1,m ) = q3 ,

(69)

we have the following: (1) restricted to F(H L ), the solution z satisfies the discrete equation of Toda type (13) on the hexagonal lattice 3 X k=1

(2)

Ak



 1 1 + = 0; z − zk z − z k+3

(70)

restricted to V (H L ), the solution z satisfies 3 X k=1

Ak

1 = 0, z − zk

(71)

562

BOBENKO and HOFFMANN

where Ai = 1i+2 − 1i+1 with 1i defined through qi = 1i+2 /1i+1 . Conversely, given q1 , q2 , q3 ∈ C with q1 q2 q3 = 1, a solution z to (70) on F(H L ) ˆ satisfying (69). and z 0,0,0 , there is a unique extension z : V (T L ) → C Given q1 , q2 , q3 ∈ C with q1 q2 q3 = 1, a solution z to (71) on V (H L ) and ˆ satisfying (69). z 1,0,0 , there is a unique extension z : V (T L ) → C Proof First, (67) immediately shows that (71) is equivalent to having constant S as described in Section 5. Again we distinguish the sublattices notationally by denoting points associated with elements of V (H L ) with wi , and points from F(H L ) with z i . If the crossratios and neighboring points of a z ∈ F(H L ) are labelled as shown in Figure 14, equation (67) gives six equations: 1 z − z1 1 (q1 − 1) z − z4 1 (q2 − 1) z − z2 1 (q2 − 1) z − z5 1 (q3 − 1) z − z3 1 (q3 − 1) z − z6 (q1 − 1)

1 z − w1 1 = q1 z − w4 1 = q2 z − w2 1 = q2 z − w5 1 = q3 z − w3 1 = q3 z − w6 = q1

− − − − − −

1 , z − w6 1 , z − w3 1 , z − w1 1 , z − w4 1 , z − w2 1 . z − w5

(72)

To prove the first statement, we take a linear combination of the equations (72). Namely, a times the first two plus b times the second two plus c times the third two. It is easy to see that there is a choice for a, b, and c that makes the right-hand side vanish if and only if q1 q2 q3 = 1. If q1 = 13 /12 , q2 = 11 /13 , and q3 = 12 /11 , choose a = 12 , b = 13 , and c = 11 . The remaining equation is (70). To prove the third statement, we note that, given w1 , we can compute w2 through w6 from the second through sixth equations of (72), but the closing condition, the first equation of (72), is then equivalent to (70). The proofs of the second and fourth statements are literally the same if we choose z i = z i+3 and wi = wi+3 .

HEXAGONAL CIRCLE PATTERNS

563

Appendix C. Proofs of Theorems 8.1 and 8.2 Proof of Theorem 8.1 Let 8k,`,m (µ) be a solution to (43), (46) with some µ-independent matrices (n) Ck,`,m , Bk,`,m . The determinant identity det 8k,`,m (µ) = (1 − µ11 )k (1 − µ12 )` (1 − µ13 )m det 80,0,0 (µ) implies that tr Ak,`,m (µ) =

k ` m + + + a(µ) µ − 1/11 µ − 1/12 µ − 1/13

with a(µ) independent of k, `, m. Without loss of generality, one can assume a(µ) = 0; that is, (1) (2) (3) tr Bk,`,m = k, tr Bk,`,m = `, tr Bk,`,m = m. R This can be achieved by the change 8 7→ exp(−1/2 a(µ) dµ)8. The compatibility conditions of (48) and (49) read d L (1) = Ak+1,`,m L (1) − L (1) Ak,`,m , dµ d L (2) = Ak,`+1,m L (2) − L (2) Ak,`,m , dµ d L (3) = Ak,`,m+1 L (3) − L (3) Ak,`,m . dµ

(73)

The principal parts of these equations in µ = 1/11 imply     1 f1 1 f1 (1) = 1  B (1) , f 1 = z k+1,`,m − z k,`,m , Bk+1,`,m  1 k,`,m 1 1 f1 f1     1 f2 1 f2 (1)  =  12  B (1) , f 2 = z k,`+1,m − z k,`,m , Bk,`+1,m  12 k,`,m 1 1 f 2 11 f 2 11     1 f3 1 f3 (1)  =  13  B (1) , f 3 = z k,`,m+1 − z k,`,m . Bk,`,m+1  13 k,`,m 1 1 f 3 11 f 3 11 (1)

The solution with tr Bk,`,m = k is (1)

Bk,`,m =

k − a1 z k+1,`,m − z k−1,`,m  z k+1,`,m − z k,`,m · 1

(z k+1,`,m − z k,`,m )(z k,`,m − z k−1,`,m ) z k,`,m − z k−1,`,m

 +

a1 I. 2

564

BOBENKO and HOFFMANN (2)

(3)

The same computation yields the formulas of Theorem 8.1 for Bk,`,m and Bk,`,m . To derive a formula for the coefficient Ck,`,m , let us compare 8k,`,m (µ) with the solution 9k,`,m (λ) in Theorem 7.3, more exactly, with its extension to the lattice Z3 : 9k+1,`,m (λ) = L (1) (λ)9k,`,m , 9k,`+1,m (λ) = L (2) (λ)9k,`,m , 9k,`,m+1 (λ) = L (3) (λ)9k,`,m , normalized by 90,0,0 (λ) = I . Here the matrices L (n) are given by (42):   1 λ fn (n) 2 −1/2 L (λ) = (1 − λ 1n ) . λ1n / f n 1 Consider ˜ = h(λ) 9 with



√   √  λ 0√ 1/ λ √0 9 0 λ 0 1/ λ

h(λ) = (1 − λ2 11 )k/2 (1 − λ2 12 )`/2 (1 − λ2 13 )m/2 .

˜ is a function of µ. Since it satisfies the same difference equations (48) So-defined 9 as 8(µ) and is normalized by ˜ k,`,m (µ = 0) = I, 9 we have ˜ k,`,m (µ)80,0,0 (µ). 8k,`,m (µ) = 9

(74)

˜ Moreover, 9(µ) is holomorphic in µ = 0 and, due to Theorem 7.3, equal at this point to   1 Z ˜ 9k,`,m (µ = 0) = , Z = z k,`,m − z 0,0,0 . 0 1 Taking the logarithmic derivative of (74) with respect to µ, Ak,`,m =

˜ k,`,m −1 d9 ˜ ˜ ˜ −1 9 k,`,m + 9k,`,m A0,0,0 9k,`,m , dµ

and computing its singularity at µ = 0, we get     1 Z 1 −Z Ck,`,m = C0,0,0 . 0 1 0 1 The formula for Ck,`,m in Theorem 8.1 is the general solution to this equation. Conversely, by direct computation one can check that the compatibility condi2 tions (73) with Ak,`,m computed above are equivalent to (46).

HEXAGONAL CIRCLE PATTERNS

565

Proof of Theorem 8.2 First, note that one can assume that b = 0 by applying a suitable M¨obius transformation. Next, by translating z, one can make z k,l,m = 0 for arbitrary fixed k, l, m ∈ Z. (This changes d, however.) Finally, we can assume that d = 0 or d = 1 since we can scale z. Now one can show the compatibility by using a computer algebra system like MATHEMATICA as follows: given the points z k,l,m , z k+1,l,m , z k,l±1,m , z k,l,m±1 , one can compute z k−1,l,m using the constraint (46). With the cross-ratio equations, one can now calculate all points necessary to apply the constraint for calculating z k+2,l,m and z k,l,m+2 . (In addition to z k−1,l,m−1 , one needs all points z k±1,l±1,m±1 .) Once again using the cross-ratios, one calculates all points necessary to calculate z k+1,l,m+2 with the constraint. Finally, one can check that the cross-ratio q(z k,l,m+1 , z k+1,l,m+1 , z k+1,l,m+2 , z k,l,m+2 ) is correct. Since the three directions are equivalent and since all initial data were arbitrary (i.e., symbolic), this suffices to show the compatibility. Acknowledgments. The authors thank S. I. Agafonov and Yu. B. Suris for collaboration and helpful discussions. References [A]

` ADLER, Legendre transformations on a triangular lattice, Funct. Anal. Appl. 34 V. E.

[Ag]

S. I. AGAFONOV, Embedded circle patterns with the combinatorics of the square grid

(2000), 1 – 9. MR 2001e:37101 526, 534

[AB]

[B] [BDS] [BS] [BH]

[BHS]

and discrete Painlev´e equations, Discrete Comp. Geom. 29 (2003), 305 – 319. 553 S. I. AGAFONOV and A. I. BOBENKO, Discrete Z γ and Painlev´e equations, Internat. Math. Res. Notices 2000, 165 – 193. MR 2001g:39042 526, 527, 553, 556, 557, 558 R. J. BAXTER, Exactly Solved Models in Statistical Mechanics, Academic Press, London, 1982. MR 86i:82002a 544 A. F. BEARDON, T. DUBEJKO, and K. STEPHENSON, Spiral hexagonal circle packings in the plane, Geom. Dedicata 49 (1994), 39 – 70. MR 95e:52029 526 A. F. BEARDON and K. STEPHENSON, The uniformization theorem for circle packings, Indiana Univ. Math. J. 39 (1990), 1383 – 1425. MR 92b:52038 525 A. I. BOBENKO and T. HOFFMANN, Conformally symmetric circle packings: A generalization of Doyle’s spirals, Experiment. Math. 10 (2001), 141 – 150. MR 2002a:52023 526, 527, 538, 541 A. I. BOBENKO, T. HOFFMANN, and YU. B. SURIS, Hexagonal circle patterns and integrable systems: Patterns with the multi-ratio property and Lax equations on the regular triangular lattice, Internat. Math. Res. Notices 2002, 111 – 164. CMP 1 878 802 527, 538, 558

566

[BP1] [BP2]

[CR] [DS] [H] [HS] [I] [K] [KS]

[KN] [NC] [ND]

[R] [RS] [S] [Su] [T]

BOBENKO and HOFFMANN

A. I. BOBENKO and U. PINKALL, Discrete isothermic surfaces, J. Reine Angew. Math.

475 (1996), 187 – 208. MR 97f:53004 546 , “Discretization of surfaces and integrable systems” in Discrete Integrable Geometry and Physics (Vienna, 1996), ed. A. I. Bobenko and R. Seiler, Oxford Lecture Ser. Math. Appl. 16, Oxford Univ. Press, New York, 1999, 3 – 58. MR 2001j:37128 526, 527, 530, 546 K. CALLAHAN and B. RODIN, Circle packing immersions form regularly exhaustible surfaces, Complex Variables Theory Appl. 21 (1993), 171 – 177. MR 95i:30029 T. DUBEJKO and K. STEPHENSON, Circle packing: Experiments in discrete analytic function theory, Experiment. Math. 4 (1995), 307 – 348. MR 97f:52027 525, 526 Z.-X. HE, Rigidity of infinite disc patterns, Ann. of Math. (2) 149 (1999), 1 – 33. MR 2000j:30068 526 Z.-X. HE and O. SCHRAMM, The C ∞ -convergence of hexagonal disk packings to the Riemann map, Acta Math. 180 (1998), 219 – 245. MR 99j:52051 525 A. R. ITS, “Isomonodromic” solutions of equations of zero curvature, Math. USSR-Izv. 26 (1986), 497 – 529. MR 87e:35078 549 P. KOEBE, Kontaktprobleme der konformen Abbildung, Ber. Verh. S¨achs. Akad. Wiss. Leipzig Math.-Phys. Kl. 88 (1936), 141 – 164. 525 B. G. KONOPELCHENKO and W. K. SCHIEF, Menelaus’ theorem, Clifford configurations and inversive geometry of the Schwarzian KP hierarchy, J. Phys. A 35 (2002), 6125 – 6144. CMP 1 930 775 538 I. M. KRICHEVER and S. P. NOVIKOV, Trivalent graphs and solitons, Russian Math. Surveys 54 (1999), 1248 – 1249. MR 2000k:37099 526 F. NIJHOFF and H. CAPEL, The discrete Korteweg – de Vries equation, Acta Appl. Math. 39 (1995), 133 – 158. MR 96i:39022 546 S. P. NOVIKOV and I. A. DYNNIKOV, Discrete spectral symmetries of low-dimensional differential operators and difference operators on regular lattices and two-dimensional manifolds, Russian Math. Surveys 52 (1997), 1057 – 1116. MR 99e:35029 526 B. RODIN, Schwarz’s lemma for circle packings, Invent. Math. 89 (1987), 271 – 289. MR 88h:11043 525 B. RODIN and D. SULLIVAN, The convergence of circle packings to the Riemann mapping, J. Differential Geom. 26 (1987), 349 – 360. MR 90c:30007 525 O. SCHRAMM, Circle patterns with the combinatorics of the square grid, Duke Math. J. 86 (1997), 347 – 389. MR 98a:30061 526, 530, 558 YU. B. SURIS, On some integrable systems related to the Toda lattice, J. Phys. A. 30 (1997), 2235 – 2249. MR 98h:58095 534 W. P. THURSTON, The finite Riemann mapping theorem, invited talk at the international symposium on the occasion of the proof of the Bieberbach conjecture, Purdue Univ., West Lafayette, Ind., 1985. 525

Bobenko Fakult¨at II, Institut f¨ur Mathematik, Technische Universit¨at Berlin, Strasse des 17 Juni 136, D-10623 Berlin, Germany; [email protected]

HEXAGONAL CIRCLE PATTERNS

Hoffman Fakult¨at II, Institut f¨ur Mathematik, Technische Universit¨at Berlin, Strasse des 17 Juni 136, D-10623 Berlin, Germany; [email protected]

567

A POLYTOPE CALCULUS FOR SEMISIMPLE GROUPS JARED E. ANDERSON

Abstract We define a collection of polytopes associated to a semisimple group G. Weight multiplicities and tensor product multiplicities may be computed as the number of such polytopes fitting in a certain region. The polytopes are defined as moment map images of algebraic cycles discovered by I. Mirkovi´c and K. Vilonen. These cycles are a canonical basis for the intersection homology of (the closures of the strata of) the loop Grassmannian. 1. Introduction Starting with a semisimple complex algebraic group G, we construct a collection of polytopes. The central result is a method that uses these to decompose the tensor product of two irreducible representations. Each tensor product multiplicity is the number of polytopes in a certain set. Other combinatorial descriptions of these numbers are known, notably by A. Berenstein and A. Zelevinsky [BZ] and P. Littelmann [LI]. The method here is based on the geometry of the loop Grassmannian and builds directly on the work of Mirkovi´c and Vilonen [MV]. But all algebraic geometry is deferred until Section 5 since we may state the main result (Theorem 1) without it. We do this in Section 2, and we follow with a lot of examples in Sections 3 and 4. Much may be gained from just these first sections without ever understanding what the loop Grassmannian is. Some background in geometry and representation theory is discussed in Section 5. The representation theory of G is known to be closely related to the geometry of the loop Grassmannian for the Langlands dual group (see [BD], [G]). This relationship was made more explicit with Mirkovi´c and Vilonen’s discovery of a collection of singular algebraic varieties in the loop Grassmannian, which we call MV-cycles (see [MV]). In terms of geometry, they provide a canonical basis for the intersection homology of the closure of each stratum of the loop Grassmannian. In terms of representation theory, they provide a canonical basis for each irreducible representation of G. DUKE MATHEMATICAL JOURNAL c 2003 Vol. 116, No. 3, Received 24 September 2001. Revision received 25 January 2002. 2000 Mathematics Subject Classification. Primary 14L99; Secondary 20G05. Author’s work partially supported by a National Science Foundation Graduate Research Fellowship. 567

568

JARED E. ANDERSON

Section 6 provides a definition of the polytopes as moment map images of MVcycles (Definition 5 on page 580). The rest of the paper consists mainly of the proof of Theorem 1. Section 8 contains the main geometric idea (Theorem 8) and is of interest in its own right. The last section provides a glimpse of a closely related Hopf algebra. 2. Statement of results Let G be a connected semisimple complex algebraic group of rank n. Choose a maximal torus T ⊂ G. We consider polytopes in the real n-dimensional vector space in which pictures of roots and weights are usually drawn: the dual t∗R of the Lie algebra of the split real form of T. Let R − denote the set of negative roots, and let 3− denote the semigroup they generate. The restriction of the usual partial order on the weight lattice (µ  λ means that µ − λ ∈ 3− ) gives 3− a partial order with maximum zero. The Kostant partition function K : 3− → N is defined by letting K (ν) be the number of ways of writing ν as a sum of negative roots (without regard for order). For each ν ∈ 3− there is a set Bν of size K (ν) parametrizing a set of convex polytopes; each polytope is contained in the cone R3− , with maximum point zero (highest weight vertex) and minimum point ν (lowest weight vertex). The parametrization is injective (see [AM]), so that there really are K (ν) different polytopes here, but we do not use this. Among our polytopes are (shifts of) those familiar from representation theory: the convex hull of the weights in an irreducible representation of G; conv(W · λ) denotes the convex hull of the Weyl group orbit through a weight λ. THEOREM 1 There exists a family of polytopes M V = (Pφ )φ∈B in t∗R with parameter set B graded S by 3− (B = ν∈3− Bν ) such that weight multiplicities and tensor product multiplicities may be calculated according to the following rules. (1) If Vλ is an irreducible representation of G with dominant weight λ, then the multiplicity of the weight ν in Vλ equals the number of φ ∈ Bν−λ for which Pφ + λ ⊆ conv(W · λ). (2) If Vλ and Vµ are irreducible representations of G with dominant weights λ and µ, and ν is any dominant weight, then the multiplicity of Vν in Vλ ⊗ Vµ equals the number of φ ∈ Bν−µ−λ for which Pφ +λ ⊆ conv(W ·λ)∩(conv(W · −µ) + ν).

The polytopes in M V are called MV-polytopes. The “MV” stands for Mirkovi´c and Vilonen, who discovered a collection of algebraic varieties called MV-cycles. MVpolytopes are defined as moment map images of MV-cycles. Theorem 1(1) (proved in Section 7) relies on some algebraic geometry of Mirkovi´c and Vilonen. Theorem

A POLYTOPE CALCULUS FOR SEMISIMPLE GROUPS

569

Figure 1

1(2) (proved in Section 9) depends on some new geometry in Section 8. 3. Examples of polytopes We explicitly describe the collection of polytopes M V for a few low-rank groups: SL2 , SL3 , Sp4 , SL4 . Such a description is not known for other groups, although Mirkovi´c and J. Anderson have a conjecture that inductively constructs the polytopes for any group (see [AM]). Before describing all the polytopes for these groups, we introduce a picture of eight of them (see Figure 1), which count the weight multiplicities in the adjoint representation of SL3 . The weight multiplicity is 1 at each outer vertex and 2 at the centre. To describe the polytopes, it is easiest to first introduce a commutative algebra A with basis M V . This algebra is discussed briefly in Section 10. In these examples we give ad hoc definitions of A by a natural set of generators and relations. Then a collection of monomials in the generators is defined. A polytope is associated to each generator, and then to each monomial, by taking the Minkowski sum of the factors. (The Minkowski sum of two sets A and B is the set of sums {a + b | a ∈ A, b ∈ B}.)

570

JARED E. ANDERSON

3.1. SL2 A = Z[a], and the MV-polytopes [kα, 0] correspond to monomials a k (α is the negative root in t∗R = R). 3.2. SL3 The algebra A has four generators (see Figure 2). (The negative roots are indicated.

a1 b1 roots

b2

a2 Figure 2

For each polytope, the white, highest-weight vertex is at the origin, and the black, lowest-weight vertex indicates the grading.) There is a single relation, a1 a2 = b1 +b2 , which we put in a diagram (see Figure 3). The MV-polytopes correspond to monomij j als of the form a1i b1 b2k or a2i b1 b2k , that is, those monomials in the generators such that a1 and a2 do not both occur. These are the monomials that cannot be further simplified using the relation. Again, the MV-polytopes are found by taking Minkowski sums. For example, the regular hexagon b1 b2 is the Minkowski sum of the two triangles. In j general, the monomials b1 b2k give shifts of the symmetric hexagons conv(W · λ). An arbitrary MV-polytope is one of these hexagons stretched some length in the direction of either a1 or a2 . The first few are drawn in Figure 4 on page 572. As an aside, note that the relation a1 a2 = b1 + b2 has an interpretation in terms of Minkowski sums: The Minkowski sum of the two line segments a1 and a2 is a parallelogram that equals the union of the two triangles b1 and b2 . In general, if αβ = P γ ∈M V n γ γ , then the Minkowski sum of α and β equals the union of the γ for which n γ 6= 0. (One containment is proved in [A].)

a1

b1 + b2

a2

Figure 3

b1

b2

A POLYTOPE CALCULUS FOR SEMISIMPLE GROUPS

571

3.3. Sp4 The algebra A has eight generators (see Figure 5 on page 573) and nine relations (see Figure 6 on page 573). The MV-polytopes correspond to monomials of the following j j j j j j forms: a1i b1 c3k d1l , a1i b2 c3k d1l , a2i c1 c3k d1l , a2i c2 c3k d1l , b1i c1 c3k d1l , b2i c2 c3k d1l . Equivalently, these are all monomials in the generators, no two factors of which are joined by a line in the diagram of relations. The monomials c3k d1l give shifts of the symmetric octagons conv(W · λ). The first few MV-polytopes are shown in Figure 7 on page 574. 3.4. SL4 The example of SL4 is very rich and harder to think about since the polytopes are three-dimensional. There are twelve generators (see Figure 8 on page 575). These are drawn relative to an octahedron whose front left vertex is the origin of t∗R = R3 . The three line segments a1 , a2 , a3 identify the negative simple roots. The bi are triangles; c1 , c2 are square-based pyramids; c3 , c4 are tetrahedra; d1 is an octahedron. The fifteen relations are shown in Figure 9 on page 575. As before, the collection of monomials consists of those for which no two factors are joined by a line in this diagram. Remark. The above calculations are based on the geometry of the loop Grassmannian. In the cases of Sp4 and SL4 , it is conjectural that the polytopes described are the right ones; in SL2 and SL3 this is easier to verify. The generators and relations were initially guessed by trying to find unique factorizations of MV-polytopes in terms of Minkowski sums, then partially checked by geometry in the loop Grassmannian. In the case of SL5 , the generators, and enough information to easily compute the relations, may be found in [A]; this case gives counterexamples to several conjectures one might be tempted to make based on the examples here—in particular, it is no longer true that every generator fits inside the Weyl polytope for a fundamental representation or that every relation gives the product of two generators as the sum of only two monomials. In the low-rank examples we have described, these generators and relations were known, in a different context, to A. Zelevinsky and his collaborators and were used by them as motivation for their study of cluster algebras (see [FZ]).

572

JARED E. ANDERSON

a12

a12 b1

a1

a12 b2

1

a1 b1 a1 b2

a1 b12

a1 b1 b2 b12 b2

b1

a1 b22

a2 b12

b1 b2 b2 2

b13

a2 b1 b23

b2

a2 b2

a2 b22

b1 b22

a22 a2 b12

a2 b1 b2 Figure 4

A POLYTOPE CALCULUS FOR SEMISIMPLE GROUPS

573

a1 b1 b2 roots

d1

c1

a2

c3

c2 Figure 5

c1

b2 b1 c3 + a2 d1 b12 + d1

a22 d1 + c32

a1

c2 + c3 b1 + b2

b22 + d1 c2

a2

c3

a1 c3 + d1

d1

c1 + c3 b2 c3 + a2 d1

Figure 6

b1

574

JARED E. ANDERSON

a12

a1 b2 a1

a 1 b1

b12 b22

a1 c3

b1 b2

1

d1 b1 c1 b1 c3 c1

a2 d1

b2 c2 b2 c3

Figure 7

a2 c3 c 2

A POLYTOPE CALCULUS FOR SEMISIMPLE GROUPS

roots

relative to octahedron

their convex hull

575

a1

a2

a3

b1

b2

b3

b4

c1

c2

c3

c4

d1 Figure 8

a1 d1 + b1 c4

a3 d1 + b4 c3 b2

b3

c2 b1 b4 + d1

c1 + c4

c1 + c3 a1 a3 d1 + c3 c4 a1

b1 + b3

a2

b2 + b4

a3

c2 + c3 b2 b3 + d1 a3 d1 + b2 c4

d1

c4

a2 c4 + d1 c2 + c4

b4

c3

c1

a2 c3 + d1 a1 d1 + b3 c3 Figure 9

b1

576

JARED E. ANDERSON

4. Examples of multiplicity calculations We use the MV-polytopes for SL3 in two examples, illustrating the two parts of Theorem 1. Example. SL3 weight multiplicity calculation This example illustrates Theorem 1(1). The hexagon in Figure 10 corresponds to an irreducible representation of SL3 of highest weight λ. Suppose we want to know the weight multiplicity at the indicated weight ν. Of the four MV-polytopes parametrized by Bν−λ , three are contained in the hexagon, and one is not. (Its bottom right corner sticks out.) So the weight multiplicity is 3.

λ ν

contains

but not

b13

b12 b2

b1 b22

b23

Figure 10

Example. SL3 tensor product multiplicity calculation This example illustrates Theorem 1(2). Suppose we want to decompose the tensor product of the two irreducible SL3 -representations with highest weights λ and µ indicated in Figure 11. The dashed lines are the outlines of the irreducible summands that occur with nonzero multiplicity. The MV-polytopes that were counted to find the multiplicities are drawn in gray; these were the ones contained in the intersection of the bigger hexagon conv(W · λ) and the smaller translated hexagon conv(W · −µ) + ν. Seven summands have multiplicity 1, and two have multiplicity 2: a1 b1 , a1 b2 and b1 , b2 . Remark. This calculation has a flavour similar to that of the convolution of the characters of the two representations, where one computes a sum of products of weight multiplicities instead of counting polytopes. After convolving the characters, one uses the Steinberg multiplicity formula to find the irreducible summands. Our method required no knowledge of weight multiplicities, no arithmetic, and no separate calculation to extract the irreducible summands. Of course, saying that this method is more efficient than the Steinberg multiplicity formula is not saying very much. One might hope that it would provide an explicit formula for the tensor product multiplicities, but it seems to provide only an algorithm.

A POLYTOPE CALCULUS FOR SEMISIMPLE GROUPS

λ

1

a1 b1

µ

tensor

577

equals the direct sum of these irreducibles (the dashed lined Weyl polytopes)

a2

a1 b2

a1

a12

b1

b2

a12 b1 a1 b1 b2

b1 b2 Figure 11

5. Geometry in representation theory 5.1. Loop Grassmannian Our basic object of study is the loop Grassmannian (affine Grassmannian), which is an infinite-dimensional space associated to a complex algebraic group G, taken to be connected and semisimple. (For us, G is the Langlands dual of the group G whose representation theory we are interested in.) The loop Grassmannian is the quotient of groups G = G(K )/G(O ), where O = C[[t]], the ring of formal power series, and K = C((t)), its field of fractions, the ring of formal Laurent series. One can show that if we use the rings C[t, t −1 ] and C[t] instead of K and O in this definition, then we get the same set. Since the numerator G(C[t, t −1 ]) is a group of maps from the unit circle to G (let t = eiθ ), we understand the use of the word loop. There is a model of G which describes it as a set of subspaces of an infinitedimensional vector space, which explains the use of Grassmannian (see [L]). The orbits of the action of G(O ) on G by left multiplication provide a stratification of G and are indexed by the dominant coweights of G. The following is our notation for this. Fix a maximal torus T ⊆ G. (For us, T is the dual of T.) Any

578

JARED E. ANDERSON

coweight λ ∈ Hom(C∗ , T ) defines an element of T (K ) ⊆ G(K ), and hence one of G , which we denote λ. Let Gλ = G(O )λ. Any point in G is in some Gλ , and this λ is determined up to the action of the Weyl group. Note also that the coweights, viewed as elements of T (K ), act on G . Each stratum Gλ is a complex vector bundle over a flag manifold G(C)λ for G, sometimes called the core. For a coweight λ in the interior of the positive Weyl chamber, this is the full flag manifold, but if λ lies on a Weyl chamber wall, then it is some partial flag manifold. The closure Gλ of a stratum consists of the union of all Gµ with µ  λ, where λ and µ are dominant. Gλ is a finite-dimensional complex projective algebraic variety, and it is almost always singular (see [PS]). 5.2. Relation to representation theory It turns out that what gives information about representation theory is the intersection homology of M. Goresky and R. MacPherson [GM1], [GM2]. Work of V. Drinfeld [BD], V. Ginzburg [G], G. Lusztig, and Mirkovi´c and Vilonen [MV] shows that the intersection homology IH(Gλ ) of the closure of a stratum is canonically isomorphic to the vector space underlying the representation Vλ of the Langlands dual group G L = G. (Here λ is both a coweight of G and a weight of G.) They actually show much more; the category of G(O )-equivariant perverse sheaves on G can be given a natural tensor product and is equivalent to the tensor category of representations of G. 5.3. MV-cycles Mirkovi´c and Vilonen discovered a canonical basis of algebraic cycles for IH(Gλ ) [MV]. These MV-cycles are projective varieties in the loop Grassmannian. IH(Gλ ) is represented by those MV-cycles contained in Gλ but not contained in any Gµ with µ ≺ λ. To define MV-cycles, we must first make some choices. Fix opposite Borel subgroups of G which intersect in the maximal torus T . Denote by N and N − their unipotent radicals. We choose the positive roots to be roots in N − . Then for any coweights λ, µ, we let Sλ = N (K )λ and Tµ = N − (K )µ. All N (K )- and N − (K )-orbits are uniquely of this form. These orbits have both infinite dimension and infinite codimenS S sion in G . There are simple closure relations: Sλ = ξ λ Sξ and Tµ = ηµ Tη (see [MV]). Definition 2 Let Gλ be the closure of a stratum of the loop Grassmannian, where λ is chosen dominant. Let ν be a coweight of G with ν ∈ Gλ . The MV-cycles for Gλ at ν, relative to N , are the irreducible components of Sν ∩ Gλ . Equivalently (as we prove below), they are those irreducible components of Sν ∩ Tλ contained in Gλ .

A POLYTOPE CALCULUS FOR SEMISIMPLE GROUPS

579

The equivalence of the two definitions is a consequence of the dimension calculations of [MV]; both Sν ∩ Tλ and Sν ∩ Gλ have pure (complex) dimension equal to the height of λ − ν. PROPOSITION 3 The two definitions are equivalent.

Proof Suppose A ∈ Irr(Sν ∩ Tλ ) (i.e., A is an irreducible component of this variety) and S S A ⊆ Gλ . Evidently, A ⊆ Sν ∩ Gλ . Since Sν = ξ ν Sξ and Gλ = dominant ηλ Gη , we see that Sν ∩ Gλ = Sν ∩ Gλ ∪ X , where every component of X has dimension strictly less than height (λ − ν), the dimension of A. So A ∈ Irr(Sν ∩ Gλ ). Conversely, suppose A ∈ Irr(Sν ∩ Gλ ). According to [MV], dim(Gλ ∩ Tλ ) = S dim(Gλ ); therefore Gλ ⊆ Tλ . So A ⊆ Sν ∩ Tλ . Since Sν = ξ ν Sξ and Tλ = S ηλ Tη , we see that Sν ∩ Tλ = Sν ∩ Tλ ∪ Y , where every component of Y has dimension strictly less than height(λ − ν), the dimension of A. (In fact, we believe Y is empty.) So A ∈ Irr(Sν ∩ Tλ ) and clearly A ⊆ Gλ . When ν = λ, there is one MV-cycle, the point λ. When ν = w0 λ (where w0 is the longest element of the Weyl group), there is also one MV-cycle, the whole variety Gλ . In general, the number of irreducible components of Sν ∩ Gλ is the dimension of the weight space ν in the irreducible representation Vλ (see [MV]). 6. The moment map and MV-polytopes The moment map 8, for the action of the torus T on G , is a map from G to Lie(T )∗ . We define 8 as the restriction of the moment map on a projective space. Let L be the determinant bundle on G (see [PS]), and let 0(G , L ) be the vector space of global sections. Then G naturally embeds in the projective space P(V ) where V = 0(G , L )∗ by mapping x ∈ G to the point determined by the line in V dual to the hyperplane {s ∈ 0(G , L ) | s(x) = 0}. The action of the torus T on V decomposes L ∗ it into eigenspaces: V = ν∈X ∗ (T ) Vν , where X (T ) denotes the weights. Choose an inner product on V which is invariant under the action of the maximal compact P subgroup of T , so that this decomposition is orthogonal. Then, given v = vν ∈ V , P we define 8([v]) = (|vν |2 /|v|2 )ν, the usual moment map on a projective space. (Another definition may be found in [AP].) Since moment map images have to do with the representation theory of G, which is usually pictured in the real subspace t∗R of Lie(T)∗ , we want 8 to map to Lie(T)∗ not Lie(T )∗ . By duality, Lie(T )∗ is canonically identified with Lie(T). The Killing

580

JARED E. ANDERSON

form identifies Lie(T) with Lie(T)∗ . We use these identifications to view Lie(T)∗ as the codomain of 8. 4 The T -fixed points in G are the ν (where ν is a coweight of G) and 8(ν) = ν. Those in Gλ are the W · λ. Those in Gλ are the ν where ν ∈ conv(W · λ) ∩ (λ + root lattice). If X is a one-dimensional torus orbit, then 8(X ) is a line segment in a root direction joining two weights. If A is an MV-cycle, then 8(A) is a convex polytope. It is the intersection of a finite number of half-spaces, each lying on one side of an affine hyperplane spanned by roots. We have 8(Gλ ) = conv(W · λ). If X ⊆ G is any compact torus-invariant variety and η is any coweight, then 8(ηX ) = η + 8(X ).

PROPOSITION

(1)

(2) (3)

(4) (5)

Proof (1) It is easy to check that any ν is a fixed point. That there are not others follows, for instance, from decomposing G into N (K ) orbits. For a coweight ν, we have v = vν so that 8(ν) = ν. The last statement follows by the closure relations for strata. (2) 8(X ) is certainly a line segment joining two weights (see [GM3]). That it lies in a root direction follows from viewing G as a partial flag variety and knowing the T -invariant curves in a flag variety (see [C]). (3) The moment map image of any compact irreducible torus-invariant variety is the convex hull of the images of its T -fixed points (see [B], [GM3]). The second statement is proved in [A] and is not used here. (4) 8(Gλ ) is the convex hull of the images of its fixed points. (5) For each fixed point ξi of X , we have 8(ηξi ) = 8(η + ξi ) = η + ξi = η + 8(ξi ). The ηξi are the fixed points of ηX . The statement follows since 8(X ) and 8(ηX ) are the convex hulls of the 8(ξi ) and 8(ηξi ), respectively.

Definition 5 For each ν ∈ 3− , let Bν = Irr(Sν ∩ T0 ), and for each irreducible component φ in Bν , let Pφ be its moment map image 8(φ). Define M V = (Pφ )φ∈B , where B = S ν∈3− Bν .

A POLYTOPE CALCULUS FOR SEMISIMPLE GROUPS

581

7. Proof of weight multiplicity calculation (Theorem 1(1)) Here we need the larger torus action of C∗ × T on G = G(K )/G(O ): The first factor acts on the indeterminate that occurs in K , and the T acts just as before. Strata and MV-cycles are preserved by this action. The following lemma lets us reduce a containment of varieties to a containment of polytopes. LEMMA 6 Let X be a ( C∗ × T )-orbit on G . Then X ⊆ Gλ for some λ ∈ X (not necessarily dominant).

Proof We must have X ⊆ G for some . Fix a point x ∈ X . G is a vector bundle over the core G(C), and the action of small z ∈ C∗ (the first factor in C∗ × T ) on x sends it close to the core. Choosing a sequence of such z n converging to zero, we can construct a point y = lim z n x contained in both X and the core. Then acting by the second factor T allows us to move y arbitrarily close to some fixed point λ in the core (and still in X ); such points are the Weyl translates of . So G = Gλ , and we have X ⊆ Gλ with λ ∈ X . 7 Let ξ be any fixed point in the closure of the stratum Gη , where η is chosen dominant. An irreducible component A of Sξ ∩ Tη is an MV-cycle if and only if its moment map image 8(A) is contained in 8(Gη ). PROPOSITION

Proof If A is an MV-cycle, then it is a component of Sξ ∩ Gη , which is contained in Gη . So 8(A) is contained in 8(Gη ). To see the other direction, assume 8(A) ⊆ 8(Gη ). Suppose a ∈ A. Then a ∈ X for some (C∗ × T )-orbit X ⊆ A. By Lemma 6, X ⊆ Gλ , where λ ∈ 8(X ) ⊆ 8(A) ⊆ 8(Gη ). By the closure relations for strata, this implies Gλ ⊆ Gη . Hence a ∈ Gη and we have A ⊆ Gη . So A is an MV-cycle for Gη . Proof of Theorem 1(1) According to [MV], the weight multiplicity at weight ν in an irreducible representation Vλ equals the number of MV-cycles at weight ν, that is, the number of components of Sν ∩ Tλ that are contained in Gλ . By Proposition 7, this is the number of A ∈ Irr(Sν ∩ Tλ ) such that 8(A) is contained in 8(Gλ ) = conv(W · λ). But this is the same as the number of A ∈ Irr(Sν−λ ∩ T0 ) such that 8(A) + λ is contained in

582

JARED E. ANDERSON

conv(W · λ). Therefore the weight multiplicity is the number of φ ∈ Bν−λ such that Pφ + λ ⊆ conv(W · λ). 8. Fibers of the convolution map This section describes the geometric idea that, when translated into the language of polytopes, yields the tensor product calculation of Theorem 1. Given two stratum closures Gλ , Gµ of the loop Grassmannian, we recall the construction of their twisted ˜ Gµ and a convolution map π from this to Gλ+µ . We show that the relevant product Gλ × irreducible components of the fibers of this map are MV-cycles. 8.1. Relative position convolution We describe the relative position convolution of the closures of two strata in the loop Grassmannian. Given two points aG(O ) and bG(O ) in G , one can ask, What is the relative position of bG(O ) with respect to aG(O )? By definition, this means, In what stratum is a −1 bG(O )? So, given closures of strata Gλ and Gµ , we can define an algebraic variety often called the twisted product: 
˜ Gµ = aG(O ), bG(O ) ∈ Gλ × Gλ+µ a −1 bG(O ) ∈ Gµ . Gλ × We are interested in this variety because of an isomorphism ˜ Gµ ) IH(Gλ ) ⊗ IH(Gµ ) ∼ = IH(Gλ × due to [BD], [G], and [MV]. IH means the global intersection homology. It always has coefficients in the trivial local system since each stratum is simply connected (see [BD]). ˜ Gµ maps to Gλ+µ by projection π onto the second factor, a stratified map Gλ × of algebraic varieties. The map is known to be semismall (see [MV]), which means that the dimension of a fiber over a stratum Gν is not larger than half the codimension of the stratum in Gλ+µ . Because of this, the decomposition theorem of A. Beilinson, J. Bernstein, and P. Deligne [BBD] has a particularly simple form in this case: M ˜ Gµ ) ∼ IH(Gλ × Fν ⊗ IH(Gν ). = Gν ⊆Gλ+µ

Here we take Fν to be the vector space spanned by the fundamental classes of each component of the fiber over ν ∈ Gν which has maximum possible complex dimension—in this case the height of λ + µ − ν. These are called the relevant components since they are the ones that appear in the above decomposition. Combined, the above two isomorphisms relate the geometry to a tensor product L ⊕dim Fν of representations: Vλ ⊗ Vµ = Vν .

A POLYTOPE CALCULUS FOR SEMISIMPLE GROUPS

583

8.2. Fibers are MV-cycles ˜ Gµ is a subset of Gλ × Gλ+µ which maps to Gλ+µ by Since the twisted product Gλ × projection onto the second factor, we may view any fiber as a subset of the first factor. Viewed as such, we show that the relevant components of the fiber at ν ∈ Gλ+µ are MV-cycles for Gλ . THEOREM 8 ˜ Gµ → Gλ+µ , and ν  λ + µ dominant, the relevant irreducible For the map π : Gλ × components of the fiber π −1 (ν) are MV-cycles for Gλ at weight ν − µ. They are precisely those that are also contained in ν G−µ . If A is such a cycle, then ν −1 A is an MV-cycle for G−µ relative to the opposite choice of unipotent subgroup.

Proof As mentioned above, we view the fiber as a subset of Gλ :  π −1 (ν) = π −1 (ν G(O )) ∼ = aG(O ) ∈ Gλ a −1 ν G(O ) ∈ Gµ  = aG(O ) ∈ Gλ ν −1 aG(O ) ∈ G−µ = Gλ ∩ ν G−µ . We claim that Gλ ∩ ν G−µ = (Gλ ∩ Sν−µ ) ∩ ν(G−µ ∩ T−ν+λ ).

(1)

Obviously, the left side contains the right side. The reverse containment is because ν G−µ ⊆ Sν−µ and Gλ ⊆ νT−ν+λ . Let us check the first of these; the second is similar. We need the following closure relations for strata and for unipotent orbits: Gξ ⊆ Gη

or, equivalently, Gξ ⊆ Gη

iff ξ  η (ξ, η dominant)
iff ξ  η (ξ, η antidominant)

and Sξ ⊆ Sη iff ξ  η (ξ, η any weights). S So Sν−µ = Sν−δ = ν S−δ = ν S−δ = ν S−µ , each union taken over δ  µ. Therefore it suffices to show G−µ ⊆ S−µ . Suppose x ∈ G−µ . Then x ∈ G− for some   µ; here −µ, − are antidominant. Now, x ∈ Sδ for some δ ∈ G− since x lies in some MV-cycle for G− [MV], so δ  −  −µ. Hence Sδ ⊆ S−µ , and x ∈ S−µ , as required. Now, MV-cycles for Gλ at weight ν − µ are the irreducible components of Gλ ∩ Sν−µ ; similarly, MV-cycles for G−µ at weight −ν + λ relative to the opposite Borel are components of G−µ ∩ T−ν+λ . These sets may be smaller than the sets S

S

584

JARED E. ANDERSON

Gλ ∩ Sν−µ and G−µ ∩ T−ν+λ in equation (1), but only by lower-dimensional compoS S nents. For instance, if we write Gλ = Gλ0 (λ0  λ dominant) and Sν−µ = Sν−µ0 (µ0  µ), we see that [ [ Gλ ∩ Sν−µ = Gλ0 ∩ Sν−µ0 = Gλ0 ∩ Sν−µ0 . λ0 ,µ0

λ0 ,µ0

(To see the second equality, note that each closure in the third expression is a subset of the first expression.) The dimension of Gλ0 ∩ Sν−µ0 is the height of λ0 + µ0 − ν, which is strictly smaller than the height of λ + µ − ν unless λ = λ0 , µ = µ0 . Therefore, Gλ ∩ Sν−µ equals Gλ ∩ Sν−µ , possibly together with some lower-dimensional MVcycles. We have shown that π −1 (ν) equals the intersection of Gλ ∩ Sν−µ and ν(G−µ ∩ T−ν+λ ), possibly together with some components of dimension less than λ + µ − ν. All statements in the theorem are proved. 9. Proof of tensor product multiplicity calculation (Theorem 1(2)) The preceding section gave a geometric interpretation of the decomposition of the tensor product of two irreducible representations Vλ and Vµ into irreducibles: the tensor product multiplicities are the dimensions of the Fν . Theorem 8 suggests a method for their calculation: Count the number of MV-cycles in each fiber. Theorem 1(2) describes this count in terms of moment map images. For another point of view on using the loop Grassmannian to decompose tensor products, see [BG], where a construction of Kashiwara’s crystal bases is given. PROPOSITION 9 Let λ, µ, ν be dominant weights with ν  λ + µ. An irreducible component A of Sν−µ ∩ Tλ is contained in Gλ ∩ ν G−µ if and only if its moment map image 8(A) is contained in 8(Gλ ) ∩ (8(G−µ ) + ν).

Proof Obviously, containment of the varieties implies containment of the moment map images. Conversely, if 8(A) ⊆ 8(Gλ ) ∩ (8(G−µ ) + ν), then 8(A) ⊆ 8(Gλ ) with A ⊆ Sν−µ ∩ Tλ , and 8(ν −1 A) ⊆ 8(G−µ ) with ν −1 A ⊆ S−µ ∩ T−ν+λ . Proposition 7 applied twice (once for the opposite unipotent) implies that A is contained in Gλ and in ν G−µ , as required. Proof of Theorem 1(2) By Theorem 8, the multiplicity with which Vν occurs in Vλ ⊗ Vµ equals the number of MV-cycles for Gλ at weight ν − µ contained in ν G−µ . This is the number of irreducible components of Sν−µ ∩ Tλ contained in Gλ ∩ ν G−µ . By Propo-

A POLYTOPE CALCULUS FOR SEMISIMPLE GROUPS

585

sition 9, this is the number of A ∈ Irr(Sν−µ ∩ Tλ ) such that 8(A) is contained in 8(Gλ ) ∩ (8(G−µ ) + ν) = conv(W · λ) ∩ (conv(W · −µ) + ν). This is the same as the number of A ∈ Irr(Sν−µ−λ ∩ T0 ) such that 8(A) + λ is contained in conv(W · λ) ∩ (conv(W · −µ) + ν). Therefore the tensor product multiplicity is the number of φ ∈ Bν−µ−λ such that Pφ + λ ⊆ conv(W · λ) ∩ (conv(W · −µ) + ν). Remark. The two parts of Theorem 1 suggest that weight multiplicities bound tensor product multiplicities. This is known, but MV-cycles provide a simple geometric explanation. 10 Suppose that λ, µ are dominant weights, and suppose that δ  0 is such that λ+µ−δ is dominant. Then the multiplicity of Vλ+µ−δ in Vλ ⊗ Vµ is less than or equal to the multiplicity of the weight λ − δ in Vλ (which is less than or equal to K (−δ) by Kostant’s formula). Moreover, these bounds are sharp in the sense that, given δ, if λ and µ are chosen sufficiently large, then for all 0    δ with λ + µ −  dominant, the multiplicity of Vλ+µ− in Vλ ⊗ Vµ exactly equals K (−). THEOREM

Proof The multiplicity of the weight λ − δ in Vλ equals the number of φ ∈ B−δ for which Pφ + λ ⊆ conv(W · λ) (Theorem 1(1)). The multiplicity of Vλ+µ−δ in Vλ ⊗ Vµ equals the number of these that are also contained in conv(W · −µ) + λ + µ − δ (Theorem 1(2)). We have sharpness because, given δ, we can choose λ and µ large enough that 8(Gλ ) ∩ (8(G−µ ) + λ + µ − δ) = 8(Sλ−δ ∩ Tλ ) and that this contains only dominant weights. 10. Hopf algebra of MV-cycles We close with a brief discussion of some related topics. This discussion is mostly conjectural, and these topics are discussed in much more detail in [A]. There we define a product on A = span(M V ) and give a conjectural definition of a coproduct. Conjecturally, A is isomorphic to the Hopf algebra of polynomial functions on the unipotent radical of a Borel subgroup of G. Another loop Grassmannian approach to the (dual) Hopf algebra may be found in [FFKM]. The product in A is defined by a deformation of varieties over a curve, using an idea of Drinfeld’s. There seem to be canonical generators and relations (described for some low-rank groups in Section 3), but little is understood about them. Positive integer coefficients appear in the relations because they are multiplicities of irreducible components. The coproduct 1 is not well understood; its definition relies on the following.

586

JARED E. ANDERSON

If X is an MV-cycle in the fiber variety π −1 (ν) of Section 8, then ((idA ⊗ S) ◦ 1)(X ) ∈ IH(Gλ ) ⊗ IH(Gµ ) corresponds to X ⊗ 1 ∈ Fν ⊗ IH(Gν ) under the canonical isomorphisms discussed there; in terms of representation theory, this is the highestweight vector of the summand corresponding to X . Here S is the antipode and idA is the identity map on A . As an example, the coproduct and antipode for Sp4 are given in Table 1. (It is only necessary to specify them on the generators since these functions are multiplicative.) Table 1

Name x 1 a1 a2 b1 b2 c1 c2 c3 d1

Antipode S(x) 1 −a1 −a2 b2 b1 −c2 −c1 −c3 d1

Coproduct 1(x) 1⊗1 a1 ⊗ 1 + 1 ⊗ a1 a2 ⊗ 1 + 1 ⊗ a2 b1 ⊗ 1 + a1 ⊗ a2 + 1 ⊗ b1 b2 ⊗ 1 + a2 ⊗ a1 + 1 ⊗ b2 c1 ⊗ 1 + 2b1 ⊗ a2 + a1 ⊗ a22 + 1 ⊗ c1 c2 ⊗ 1 + a22 ⊗ a1 + 2a2 ⊗ b2 + 1 ⊗ c2 c3 ⊗ 1 + b2 ⊗ a2 + a2 ⊗ b1 + 1 ⊗ c3 d1 ⊗ 1 + c1 ⊗ a1 + 2b1 ⊗ b2 + a1 ⊗ c2 + 1 ⊗ d1

Just as the product in A corresponds to Minkowski sum of polytopes, the coproduct has a (conjectural) interpretation in terms of polytopes. Write 1(x) = P ki j xi ⊗ x 0j , where each xi is an MV-cycle relative to N and each x 0j is an MVcycle relative to N − . If ki j 6= 0, then (1) xi and x 0j are contained in x; (2) xi and x 0j are associated to the same weight in x, and this is their only point of intersection; (3) ki j is a positive integer. Figure 12 illustrates this for 1(c1 ) = c1 ⊗ 1 + 2b1 ⊗ a2 + a1 ⊗ a22 + 1 ⊗ c1 . What are the meanings of the ki j and why are they positive? If x is the largest MVcycle in a stratum, they seem to give the intersection form in intersection homology (see [A]). If not (as in the above example), their meaning is mysterious.

A POLYTOPE CALCULUS FOR SEMISIMPLE GROUPS

c1

1

b1

a2

587

a1

a22

1

c1

Figure 12

Acknowledgments. I thank my thesis advisor, R. MacPherson, and also D. Nadler and I. Mirkovi´c, for many discussions on geometry in representation theory and for significant ideas that went into this research. I also thank T. Braden, J. Conway, D. Gaitsgory, A. Knutson, and K. Vilonen for discussions and useful suggestions. References [A]

[AM] [AP]

[BBD]

[BD]

[BZ]

[BG] [B]

[C]

J. ANDERSON, On Mirkovi´c and Vilonen’s Intersection Homology Cycles for the Loop

Grassmannian, Ph.D. Thesis, Princeton University, Princeton, 2000. 571, 580, 585, 586 J. ANDERSON and I. MIRKOVIC´ , Crystal Graphs via Polytopes, in preparation. 568, 569 M. ATIYAH and A. PRESSLEY, “Convexity and loop groups” in Arithmetic and Geometry, II, Progr. Math. 36, Birkh¨auser, Boston, 1983, 33 – 63. MR 85e:22026 579 A. A. BEILINSON, J. BERNSTEIN, and P. DELIGNE, “Faisceaux pervers” in Analyse et Topologie sur les Espaces Singuliers (Luminy, 1981), I, Ast´erisque 100, Soc. Math. France, Montrouge, 1982, 5 – 171. MR 86g:32015 582 A. BEILINSON and V. DRINFELD, Quantization of Hitchin’s integrable system and Hecke eigensheaves, preprint, 2000, http://zaphod.uchicago.edu/˜benzvi 567, 578, 582 A. BERENSTEIN and A. ZELEVINSKY, Tensor product multiplicities, canonical bases and totally positive varieties, Invent. Math. 143 (2001), 77 – 128. MR 2002c:17005 567 A. BRAVERMAN and D. GAITSGORY, Crystals via the affine Grassmannian, Duke Math. J. 107 (2001), 561 – 575. MR 2002e:20083 584 M. BRION, “Sur l’image de l’application moment” in Seminaire d’alg`ebre Paul Dubreil et Marie-Paule Malliavin (Paris, 1986), Lecture Notes in Math. 1296, Springer, Berlin, 1987, 177 – 192. MR 89i:32062 580 J. B. CARRELL, The Bruhat Graph of a Coxeter Group, a Conjecture of Deodhar, and Rational Smoothness of Schubert Varieties, Proc. Sympos. Pure Math. 56, Part 1,

588

JARED E. ANDERSON

Amer. Math. Soc., Providence, 1994, 53 – 61. MR 95d:14051 580 [DM]

P. DELIGNE and J. MILNE, “Tannakian categories” in Hodge Cycles, Motives, and

Shimura Varieties, Lecture Notes in Math. 900, Philos. Stud. Ser. Philos. 20, Springer, Berlin, 1982, 101 – 228. MR 84m:14046 [FFKM] B. FEIGIN, M. FINKELBERG, A. KUZNETSOV, and I. MIRKOVIC´ , Semiinfinite flags, II: Local and global intersection cohomology of quasimaps’ spaces, preprint, arXiv:alg-geom/9711009 585 [FZ] S. FOMIN and A. ZELEVINSKY, Cluster algebras, I: Foundations, J. Amer. Math. Soc. 15 (2002), 497 – 529 CMP 1 887 642 571 [G] V. GINZBURG, Perverse sheaves on a loop group and Langlands’ duality, preprint, arXiv:alg-geom/9511007 567, 578, 582 [GM1] M. GORESKY and R. MACPHERSON, Intersection homology theory, Topology 19 (1980), 135 – 162. MR 82b:57010 578 [GM2] , Intersection homology, II, Invent. Math. 72 (1983), 77 – 129. MR 84i:57012 578 [GM3] , “On the topology of algebraic torus actions” in Algebraic Groups (Utrecht, 1986), Lecture Notes in Math. 1271, Springer, Berlin, 1987, 73 – 90. MR 89a:14064 580 [LI] P. LITTELMANN, A Littlewood-Richardson rule for symmetrizable Kac-Moody algebras, Invent. Math. 116 (1994), 329 – 346. MR 95f:17023 567 [L] G. LUSZTIG, “Singularities, character formulas, and a q-analog of weight multiplicities” in Analysis and Topology on Singular Spaces (Luminy, 1981), II, III, Ast´erisque 101 – 102, Soc Math. France, Montrouge, 1983, 208 – 229. MR 85m:17005 577 [M] R. MACPHERSON, Intersection homology and perverse sheaves, unpublished lecture notes, 1991. [MV] I. MIRKOVIC´ and K. VILONEN, Perverse sheaves on affine Grassmannians and Langlands duality, Math. Res. Lett. 7 (2000), 13 – 24. MR 2001h:14020 567, 578, 579, 581, 582, 583 [PS] A. PRESSLEY and G. SEGAL, Loop Groups, Oxford Math. Monogr., Oxford Univ. Press, New York, 1986. MR 88i:22049 578, 579

Department of Mathematics, University of Pittsburgh, Pittsburgh, Pennsylvania 15260, USA; [email protected]