Special Subvarieties Arising from Families

of Cyclic Covers of the Projective Line

Dedicated to Frans Oort on the occasion of his 75th birthday

Ben Moonen

Received: July 14, 2010 Revised: September 2, 2010

Communicated by Peter Schneider

Abstract. We consider families of cyclic covers of ℙ^{1}, where we fix the covering group and the local monodromies and we vary the branch points. We prove that there are precisely twenty such families that give rise to a special subvariety in the moduli space of abelian varieties. Our proof uses techniques in mixed characteristics due to Dwork and Ogus.

2010 Mathematics Subject Classification: 11G15, 14H40, 14G35 Keywords and Phrases: Special subvarieties, Jacobians, complex multiplication

Introduction

According to a conjecture of Coleman, if we fix a genus g ⩾ 4 and consider smooth projective curves C over ℂ of genus g, there should be only finitely many such curves, up to isomorphism, such that Jac(C) is an abelian variety of CM type. This conjecture is known to be false for g ∈ {4, 5, 6, 7}, by virtue of the fact that for these genera there exist special subvarieties S ⊂ 𝒜_{g} (also known as subvarieties of Hodge type) of positive dimension that are contained in the (closed) Torelli locus and that meet the open Torelli locus. All known examples of such special subvarieties S arise from families of cyclic covers of ℙ^{1}. As input for this we fix integers m ⩾ 2 and N ⩾ 4, together with an N-tuple a = (a_{1}, …, a_{N}); then we consider cyclic covers C_{t} → ℙ^{1} with covering group ℤ/mℤ, branch points t_{1}, …, t_{N} in ℙ^{1}, and with local monodromy a_{i} about t_{i}. Varying the branch points we obtain an (N − 3)- dimensional closed subvariety Z = Z(m, N, a) ⊂ 𝒜_{g} given by the Jacobians of the curves C_{t}, and for certain choices of (m, N, a) it can be shown that Z is a special subvariety. The main purpose of this paper is to give a list of monodromy data (m, N, a) such that Z(m, N, a) is special, and to prove that there are no further such examples.

By construction, the Jacobians J_{t} come equipped with an action of the group ring ℤ[μ_{m}]. This action defines a special subvariety S(μ_{m}) ⊂ 𝒜_{g} that contains Z and whose dimension can be easily calculated in terms of the given monodromy data (m, N, a). We always have N − 3 ⩽ dim S(μ_{m}), and if equality holds Z = S(μ_{m}) is special. To search for triples (m, N, a) for which dim S(μ_{m}) = N − 3 is something that can be done on a computer, and it is therefore surprising that this appears not to have been done until recently. Thus, while examples in genera 4 and 6 have been given by de Jong and Noot in [6]—and in fact, these examples at least go back to Shimura’s paper [24]— examples with g = 5 and g = 7 were found only much later; see Rohde [22]. At any rate, with the help of a computer program we find twenty examples with dim S(μ_{m}) = N − 3. The main result of this paper is that this list is complete:

Main theorem. — Consider monodromy data (m, N, a) as above. Then the closed subvariety Z(m, N, a) ⊂ 𝒜_{g,ℂ} is special if and only if (m, N, a) is equivalent to one of the twenty triples listed in Table 1.

Note that if N − 3 < dim S(μ_{m}), which means that the inclusion Z ⊂ S(μ_{m}) is strict, Z could a priori still be a special subvariety. The main point of our result is that even in these cases we are able to prove that Z is not special.

To get a feeling for the difficulty of the problem, consider, as an example, the family of (smooth projective) curves of genus 8 given by y^{10} = x(x − 1)(x − t)^{2}, where t ∈ T = ℙ_{ℂ}^{1} \ {0, 1, ∞} is a parameter. The corresponding family of Jacobians J → T decomposes, up to isogeny, as a product J^{old} × J^{new} of two abelian fourfolds. (The old part comes from the quotient family u^{5} = x(x − 1)(x − t)^{2}; the new part is the family of Pryms.) Both J^{old} and J^{new} give rise to a 1-dimensional special subvariety in 𝒜_{4}, and they both admit an action by ℤ[ζ_{5}] = ℤ[ζ_{10}], with the same multiplicities on the tangent spaces at the origin. A priori it might be true that J^{old} and J^{new} are isogenous, in which case the family J → T would define a special subvariety in 𝒜_{8}. Our theorem says, in this particular example, that this does not happen. From the perspective of special points and the André-Oort conjecture, one could state the problem, in the example at hand, as follows. There are infinitely many values for t such that J_{t}^{old} is of CM type; likewise for J_{t}^{new}. Do there exist infinitely many t such that J_{t}^{old} and J_{t}^{new} are simultaneously of CM type?

Our theorem, combined with a result of Yafaev [25], implies the answer is no, at least if we assume the Generalized Riemann Hypothesis. See Corollary 3.7.

Our proof of the main theorem uses techniques from characteristic p and is based on a method that in some particular cases has already been used by de Jong and Noot in [6], Section 5. For a suitable choice of a prime number p, the Jacobians in our family will, at least generically, have ordinary reduction in characteristic p. Moreover, if we assume Z(m, N, a) ⊂ 𝒜_{g} to be special then by a result of Noot [18] we can arrange things in such a way that the canonical liftings of these ordinary reductions are again Jacobians. Already over the Witt vectors of length 2 this is a very restrictive condition that, using the results of Dwork and Ogus in [8], can be turned into something computable. In individual cases (such as the example sketched above, or the examples treated in [6]), these techniques give a rather effective method to prove that some given family of curves does not give rise to a special subvariety in 𝒜_{g,ℂ}. To do this for arbitrary data (m, N, a) is much harder, and some perseverance is required to deal with the combinatorics that is involved. It would be interesting to have a purely Hodge-theoretic proof of the main theorem; as it is, we do not even have a Hodge-theoretic method that works well in individual examples.

Let us give an overview the contents of the individual sections. In Section 1 we quickly review the notion of a special subvariety and we summarize some facts we need. In Section 2 we introduce the families of curves that we want to consider. In Section 3 we discuss the special subvariety S(μ_{m}) that contains Z(m, N, a), we give the list of data (m, N, a) for which Z(m, N, a) = S(μ_{m}) and we state the main results. Sections 4 and 5 contain the main technical tools for the proof. In Section 4 we discuss how the VHS associated with our family of curves decomposes and we give some results about the monodromy of the summands. In Section 5 we briefly review the techniques of [8] and we give some refinements that we need in order to deal with hyperelliptic families. In Sections 6 and 7, finally, we prove the main theorem, first in the case N = 4, then for N ⩾ 5. We refer to the beginning of Section 6 for a brief explanation of how the proof works.

Acknowledgements. I thank Frans Oort for his encouragement and for stimulating discussions. It is a great pleasure to dedicate this paper to him. I thank Dion Gijswijt and Fokko van de Bult for helpful suggestions related to the combinatorics involved in the proof of Lemma 7.2. Further I thank Maarten Hoeve, who wrote two Python programs (available upon request from the author) to test some of the properties we need. The results of one of these programs are used in the proof of Lemma 7.2. Finally, I thank the referee for his or her comments on the paper and for some very helpful suggestions regarding the exposition.

Notation. If x is a real number, ⟨x⟩ = x − ⌊x⌋ denotes its fractional part. For a ∈ ℤ/mℤ or a ∈ ℤ we denote by [a]_{m} the unique representative of the class a (resp. the class a mod m) in {0, …, m − 1}.

1. Some preliminaries about special subvarieties

In this section we work over ℂ. We recall the notion of a special subvariety. For further details we refer to [12], where the terminology “subvarieties of Hodge type” was used, and [20].

(1.1) Consider the moduli space 𝒜_{g,[n]} of g-dimensional principally polarized abelian varieties with a level n structure, for some n ⩾ 3. We first briefly recall the description of this moduli space as a Shimura variety. Let V_{ℤ} := ℤ^{2g} ⊂ V := ℚ^{2g}, and let Ψ: V_{ℤ} × V_{ℤ} → ℤ be the standard symplectic form. Let G := CSp(V_{ℤ}, Ψ) be the group of symplectic similitudes. Let 𝕊 := Res_{ℂ/ℝ} 𝔾_{m} be the Deligne torus, and let ℌ be the space of homomorphisms h: 𝕊 → G_{ℝ} that define a Hodge structure of type (−1, 0) + (0, −1) on V_{ℤ} for which ±(2πi) · Ψ is a polarization. The pair (G_{ℚ}, ℌ) is a Shimura datum, and 𝒜_{g} can be described as the associated Shimura variety. Concretely, if K_{n} := {g ∈ G(ℤ̂) | g ≡ 1 mod n} then 𝒜_{g,[n]}(ℂ) ≅ G(ℚ)\ℌ × G(𝔸_{fin})/K_{n}. In order to define special subvarieties, consider an algebraic subgroup H ⊂ G_{ℚ} such that

Y_{H} := {h ∈ ℌ | h factors through H_{ℝ}}

is non-empty. The group H(ℝ) acts on Y_{H} by conjugation. It can be shown (see [11], Section I.3, or [12], 2.4) that Y_{H} is a finite union of orbits under H(ℝ). We remark that the condition that Y_{H} is non-empty imposes strong restrictions on H; it implies, for instance, that H is reductive. If Y^{+} ⊂ Y_{H} is a connected component and ηK_{n} ∈ G(𝔸_{fin})/K_{n}, the image of Y^{+} × {ηK_{n}} in 𝒜_{g,[n]} is an algebraic subvariety.

(1.2) Definition. — A closed irreducible algebraic subvariety S ⊂ 𝒜_{g,[n]} is called a special subvariety if there exist Y^{+} ⊂ Y_{H} and ηK_{n} ∈ G(𝔸_{fin})/K_{n} as above such that S(ℂ) is the image of Y^{+} × {ηK_{n}} in 𝒜_{g,[n]}(ℂ).

We refer to [12] and [20] for alternative descriptions and basic properties of special subvarieties.

(1.3) As level structures play no role in what we are doing, we prefer to state our results in terms of the moduli stack 𝒜_{g}. Of course this is a stack, not a variety, but we allow ourselves to abuse terminology and speak of special subvarieties in 𝒜_{g} (over ℂ), where “special substack” would perhaps be more correct. By definition, then, a special subvariety S ⊂ 𝒜_{g} is a closed, reduced and irreducible algebraic substack such that for some (equivalently: any) n ⩾ 3 the irreducible components of the inverse image of S under the natural map 𝒜_{g,[n]} → 𝒜_{g} are special. (It is in fact enough to require that some irreducible component is special.)

(1.4) Let f: X → T be a family of g-dimensional abelian varieties over an irreducible non-singular complex algebraic variety T. Choose a principal polarization λ of X/T, and let ϕ: T → 𝒜_{g} be the resulting morphism. Whether or not the closure of ϕ(T) is a special subvariety of 𝒜_{g} only depends on the isogeny class of the generic fibre of X/T as an abelian variety over the function field of T. If we replace X/T by an isogenous family or choose a different λ, the morphism ϕ is replaced by some other morphism ϕ′: T → 𝒜_{g}, but if \overline{ϕ(T)} is special then so is \overline{ϕ′(T)}. (This is even true without the assumption that λ is a principal polarization, but we shall not need this.) Without ambiguity we may therefore say that X/T is special if \overline{ϕ(T)} ⊂ 𝒜_{g} is special for some choice of a polarization. One property we shall use is that if X/T is isogenous to Y_{1}×Y_{2} and X is special, the factors Y_{i} are both special. The converse is not true: if Y_{1}/T and Y_{2}/T are special, it is not necessarily the case that (Y_{1} × Y_{2})/T is special.

(1.5) Let f: X → T be as in 1.4. Consider the ℚ-VHS over T with fibres the first cohomology groups H^{1}(X_{t}, ℚ). Let b ∈ T (ℂ) be a Hodge-generic point for this VHS, and let M ⊂ GL(H^{1}(X_{b}, ℚ)) be the generic Mumford-Tate group of the family. Choose a principal polarization λ of X/T, let ϕ: T → 𝒜_{g} be the resulting morphism, and write Z := \overline{ϕ(T)}. Finally, let S ⊂ 𝒜_{g} be the smallest special subvariety containing Z. In order to calculate the dimension of S, it suffices to know the adjoint real group M_{ℝ}^{ad}. More precisely, if M_{ℝ}^{ad} = Q_{1} × ⋯ × Q_{r} is the decomposition of this group as a product of simple factors, dim(S) = ∑_{i=1}^{r} d(Q_{i}), where d(Q_{i}) is a contribution that only depends on the isomorphism class of the simple group Q_{i}. The only cases that are relevant for us are that

• d(Q) = 0 if Q is anisotropic (compact);

• d(Q) = pq if Q ≅ PSU(p, q);

• d(Q) = h(h + 1)/2 if Q ≅ PSp_{2h}.

2. The setup

In this section, given data (m, N, a) as in the introduction (see 2.1 below), we construct a family of cyclic covers of ℙ^{1} over some base scheme T. For later purposes we shall do this over a ring R of finite type over ℤ.

(2.1) Let m and N be integers with m ⩾ 2 and N ⩾ 2, and consider an Ntuple of positive integers a = (a_{1}, …, a_{N}) such that gcd(m, a_{1}, …, a_{N}) = 1. We further require that a_{i} ≢ 0 modulo m for all i and ∑_{i=1}^{N} a_{i} ≡ 0 modulo m. The triple (m, N, a) serves as input for our constructions. We call two such triples (m, N, a) and (m′, N′, a′) equivalent if m = m′ and N = N′ and if the classes of a and a′ in (ℤ/mℤ)^{N} are in the same orbit under (ℤ/mℤ)^{∗} × 𝔖_{N}. Here we let (ℤ/mℤ)^{∗} act diagonally by multiplication, and the symmetric group 𝔖_{N} acts by permutation of the indices. In what follows we shall usually assume N ⩾ 4 but for some arguments it is useful to allow N to be 2 or 3.

(2.2) Let (m, N, a) be a triple as in 2.1. Let R be the ring ℤ[1/m, u]/Φ_{m} with Φ_{m} the mth cyclotomic polynomial. We write ζ ∈ R for the class of u; it is a root of unity of order m. We embed R into ℂ by sending ζ to exp(2πi/m). The element ζ defines an isomorphism of R-group schemes (ℤ/mℤ)_{R} ⟶^{∼} μ_{m,R} by (b mod m) ↦ ζ^{b}. Let U ⊂ (𝔸_{R}^{1})^{N} be the complement of the big diagonals. In other words, U is the R-scheme of ordered N-tuples (t_{1}, …, t_{N}) of distinct points in 𝔸^{1}. Let B ⊂ ℙ_{U}^{2} be the projective curve over U obtained as the Zariski closure of the affine curve whose fibre over a point (t_{1}, …, t_{N}) is given by

y^{m} = (x − t_{1})^{a_{1}} ⋯ (x − t_{N})^{a_{N}} = ∏_{i=1}^{N} (x − t_{i})^{a_{i}} .

We have a μ_{m}-action on B over U by ζ · (x, y) = (x, ζ · y). The rational function x defines a morphism π_{B}: B → ℙ_{U}^{1}. There exist an open subscheme T ⊂ U, a smooth proper curve f: C → T equipped with an action of μ_{m,T}, and a μ_{m,T} -equivariant morphism ρ: C → B_{T}, such that for every point t ∈ T the morphism on fibres ρ_{t}: C_{t} → B_{t} is a normalization of B_{t}. Let π := π_{B} ∘ ρ: C → ℙ_{T}^{1} , which is a finite morphism that realizes ℙ_{T}^{1} as the quotient of C by the action of μ_{m,T}. If the context requires it, we include the data (m, N, a) in the notation, writing C = C(m, N, a) for instance. Further we write J → T for the Jacobian of C over T. If k is a field and t = (t_{1}, …, t_{N}) ∈ T (k), we can also describe π_{t}: C_{t} → ℙ_{k}^{1} as the μ_{m}-cover of ℙ_{k}^{1} with branch points t_{1}, …, t_{N} and local monodromy about t_{i} given by the element ζ^{a_{i}} ∈ μ_{m}. The assumption that gcd(m, a_{1}, …, a_{N}) = 1 implies that the fibres of f: C → T are geometrically irreducible. Let r_{i} := gcd(m, a_{i}). The Hurwitz formula gives that the fibres have genus

(2.2.1)  g = 1 + \frac{(N − 2)m − ∑_{i=1}^{N} r_{i}}{2} ,

so we obtain a morphism ψ: T → ℳ_{g} over R. Define

(2.2.2)  ϕ: T → 𝒜_{g}

to be the composition of ψ with the Torelli morphism. Up to isomorphism, the morphism ψ, and hence also ϕ, only depends on the equivalence class of the triple (m, N, a) for the equivalence relation defined in 2.1.

(2.3) Remark. — For computational purposes we have restricted our attention to the case where all branch points t_{i} are in 𝔸^{1} ⊂ ℙ^{1}, and we have not fixed any of these branch points, thereby creating some redundancy and excluding families where one of the branch points is fixed to be the point at ∞. For our main results, the only thing that really matters is the closure of the image of ϕ_{ℂ}: T_{ℂ} → 𝒜_{g,ℂ}. For instance, while the family of curves y^{10} = x(x − 1)(x − λ) is not among the families we consider (the point at ∞ being a branch point), the subvariety of 𝒜_{9} we obtain from this family is the same as the one obtained by taking m = 10, with N = 4 and a = (1, 1, 1, 7).

(2.4) Convention. — In what follows we shall in several steps replace the base scheme T by a subscheme. In such a case, it will be understood that we replace C by its restriction to the new base scheme, and we again write f: C → T for the curve thus obtained. Similarly, we retain the notation for various other objects associated with our family of curves.

(2.5) Notation. — Let M be a module over some commutative R-algebra, or a sheaf on some R-scheme, on which the group scheme μ_{m} acts. For n ∈ ℤ/mℤ we write

M_{(n)} := {x ∈ M | ζ(x) = ζ^{n} · x} ,

which in the sheaf case has to be interpreted on the level of local sections. We refer to M_{(n)} as the n-eigenspace of M. We have M = ⊕_{n∈ℤ/mℤ} M_{(n)}.

(2.6) Recall that r_{i} = gcd(m, a_{i}). Consider the μ_{m}-cover π: C_{t} → ℙ^{1} for some t ∈ T (k), where k is a field. For i ∈ {1, …, N} and n ∈ ℤ, let

l(i, n) := −1 + ⌈\frac{r_{i} − na_{i}}{m} ⌉ = ⌊\frac{−na_{i}}{m} ⌋ ,

where the second equality easily follows from the fact that r_{i} = gcd(m, a_{i}). Consider the differential forms

(2.6.1)  ω_{n,ν} := y^{n} · (x − t_{1})^{ν} · ∏_{i=1}^{N}(x − t_{i})^{l(i,n)} · dx ,

and note that these only depend on the pair (n mod m, ν). The following result is standard.

(2.7) Lemma. — Let n ∈ ℤ/mℤ with n ≠ 0. The forms ω_{n,ν} for 0 ⩽ ν ⩽ −2 + ∑_{i=1}^{N} ⟨\frac{−na_{i}}{m} ⟩ are regular 1-forms on C_{t} and they form a k-basis for H^{0}(C_{t}, Ω^{1})_{(n)}.

3. The special subvariety given by the action of the covering group

In this section we work over ℂ and we fix a triple (m, N, a) as in 2.1 with N ⩾ 4. We retain the notation introduced in the previous section, with the convention that all objects are now considered over ℂ via the chosen embedding R ↪ ℂ.

(3.1) Definition. — We let Z(m, N, a) ⊂ 𝒜_{g,ℂ} be the scheme-theoretic (or rather, stack-theoretic) image of the morphism ϕ: T → 𝒜_{g} of (2.2.2).

In other words, Z(m, N, a) is the reduced closed substack of 𝒜_{g,ℂ} with underlying topological space \overline{ϕ(T)}. We note that Z(m, N, a) only depends on the equivalence class of (m, N, a) and does not depend on the choice of the open subscheme T in 2.1. The dimension of Z(m, N, a) equals N − 3.

(3.2) Notation. — It will be convenient to write

(3.2.1)  I(m) := [(ℤ/mℤ) \ {0 mod m}]/{±1} .

Recall that ⟨x⟩ denotes the fractional part of x. For n ∈ ℤ/mℤ we define

(3.2.2)  d_{n} := \{ \begin{matrix}−1 + ∑_{i=1}^{N} ⟨\frac{−na_{i}}{m} ⟩ & if n ≢ 0, \\ 0 & if n ≡ 0, \end{matrix}

which by Lemma 2.7 is the dimension of the (n)-eigenspace of H^{0}(C_{t}, Ω^{1}), for any t ∈ T (ℂ).

(3.3) The substack Z(m, N, a) ⊂ 𝒜_{g} is contained in a special subvariety S(μ_{m}) ⊂ 𝒜_{g} determined by the action of ℤ[μ_{m}] on the relative Jacobian J → T. More precisely, S(μ_{m}) is the largest closed, reduced and irreducible substack S ⊂ 𝒜_{g} containing Z(m, N, a) such that the homomorphism ℤ[μ_{m}] → End(J/T) induced by the action of μ_{m} on C/T extends to an action of ℤ[μ_{m}] on the universal abelian scheme over S. Choose a base point b ∈ T (ℂ), and let (J_{b}, λ) be the corresponding Jacobian with its principal polarization. With (V_{ℤ}, Ψ) as in 1.1, choose a symplectic similitude σ: H_{1}(J_{b}, ℤ) ⟶^{∼} V_{ℤ}, where we equip H_{1}(J_{b}, ℤ) with its Riemann form (i.e., the polarization, in the sense of Hodge theory, that corresponds with λ). Via σ, the action of μ_{m} on J_{b} induces a structure of a ℚ[μ_{m}]-module on V = V_{ℤ} ⊗ ℚ. Consider the algebraic subgroup H ⊂ G_{ℚ} = CSp(V, Ψ) of ℚ[μ_{m}]-linear symplectic similitudes, i.e., the subgroup given by

H := GL_{ℚ[μ_{m}]}(V) ∩ CSp(V, Ψ) .

With notation as in 1.1, the image of Y_{H} ⊂ ℌ under the map

ℌ → → G(ℤ)\ℌ ≅ G(ℚ)\ℌ × G(𝔸_{fin})/G(ℤ̂) ≅ 𝒜_{g}(ℂ)

is (the set of ℂ-points of) a finite union of algebraic subvarieties of 𝒜_{g}, and the special subvariety S(μ_{m}) is the unique irreducible component of this image that contains Z(m, N, a). The dimension of S(μ_{m}) is given by

(3.3.1)  dim S(μ_{m}) = ∑ d_{−n}d_{n} + \{ \begin{matrix}\frac{d_{k}(d_{k}+1)}{2} & if m = 2k is even, \\ 0 & if m is odd, \end{matrix}

where the first sum runs over the pairs ±n ∈ I(m) with 2n ≢ 0. See [14], Section 5; see also Remark 4.6 below. In particular, as Z(m, N, a) ⊂ S(μ_{m}) we have

(3.3.2)  N − 3 ⩽ dim S(μ_{m}) .

If equality holds then Z(m, N, a) = S(μ_{m}) ⊂ 𝒜_{g} is a special subvariety; it then follows that among the Jacobians J_{t}, for t ∈ T (ℂ), there are, up to isomorphism, infinitely many Jacobians of CM type. (Here we use that on a special subvariety the CM points lie dense.)

(3.4) Inventory of examples. — Table 1 lists twenty triples (m, N, a) with N ⩾ 4 for which N − 3 = dim S(μ_{m}), so that Z(m, N, a) ⊂ 𝒜_{g,ℂ} is a special subvariety. Our assumption that N ⩾ 4 means we are only considering the cases that give rise to a special subvariety of positive dimension. The first column of the table gives a number that we assign to each example for reference. The examples are sorted first by genus, then by degree of the cover. The second column gives the genus of the curves in the family. In the next three columns we give the data (m, N, a). In most cases there is a unique a = (a_{1}, …, a_{N}) in its equivalence class such that 1 ⩽ a_{1} ⩽ ⋯ ⩽ a_{N} ⩽ m − 1 with ∑ a_{i} = 2m, and if there is a unique such representative, this is the one we list. If N = 4 there are usually two such representatives, the second one being (m − a_{4}, m − a_{3}, m − a_{2}, m − a_{1}); we list the one which lexicographically comes first. In the last column we give references to places in the literature where the example can be found. Here “S(i)” refers to example (i) in [24], “Mi” refers to example i in the appendix of [17], and dJN(1.3.i) refers to example (1.3.i) in [6]. These examples can also be found in [22]; see 4.8 below.

(3.5) Remark. — If we have a triple (m, N, a) as in 2.1 with N = 3, the associated subvariety Z = Z(m, N, a) is a special point in 𝒜_{g}.

The following theorem is the main result of this paper. It says that the list of examples in Table 1 is exhaustive.

(3.6) Theorem. — Consider data (m, N, a) as in 2.1, with N ⩾ 4. Then Z(m, N, a) ⊂ 𝒜_{g,ℂ} is a special subvariety if and only if (m, N, a) is equivalent to one of the twenty examples listed in Table 1.

By what was explained above, it only remains to be shown that Z(m, N, a) is not special if (m, N, a) is not equivalent to one of the triples in the table. In 4.9

genus  m  N  a  references
(1)  1  2  4  (1, 1, 1, 1)
(2)  2  2  6  (1, 1, 1, 1, 1, 1)
(3)  2  3  4  (1, 1, 2, 2)
(4)  2  4  4  (1, 2, 2, 3)
(5)  2  6  4  (2, 3, 3, 4)
(6)  3  3  5  (1, 1, 1, 1, 2)  S(1), M41
(7)  3  4  4  (1, 1, 1, 1)
(8)  3  4  5  (1, 1, 2, 2, 2)  S(3), M43
(9)  3  6  4  (1, 3, 4, 4)
(10)  4  3  6  (1, 1, 1, 1, 1, 1)  S(2), M23, dJN(1.3.1)
(11)  4  5  4  (1, 3, 3, 3)  S(4), dJN(1.3.2)
(12)  4  6  4  (1, 1, 1, 3)
(13)  4  6  4  (1, 1, 2, 2)
(14)  4  6  5  (2, 2, 2, 3, 3)
(15)  5  8  4  (2, 4, 5, 5)
(16)  6  5  5  (2, 2, 2, 2, 2)  S(5), M44
(17)  6  7  4  (2, 4, 4, 4)  S(6), dJN(1.3.3)
(18)  6  10  4  (3, 5, 6, 6)
(19)  7  9  4  (3, 5, 5, 5)
(20)  7  12  4  (4, 6, 7, 7)

Table 1: monodromy data that give rise to a special subvariety

we shall first prove this for m = 2. In Section 6 we shall prove the theorem for N = 4. The case N > 4 is treated in Section 7. Combining the theorem with the main result of [25] we obtain, for N = 4, the following finiteness result for the number of CM fibres.

(3.7) Corollary. — Assume the Generalized Riemann Hypothesis for CM fields. If we have a triple (m, N, a) with N = 4 and the equivalence class of this triple is not among the twenty examples listed in Table 1 then up to isomorphism there are finitely many Jacobians of CM type in the family J → T_{ℂ}.

4. Decomposition of the variation of Hodge structure

In this section we again work over ℂ. We fix a triple (m, N, a) as in 2.1 with N ⩾ 4. The corresponding family of curves C → T gives rise to a variation of Hodge structure over T (over ℂ) on which ℚ[μ_{m}] acts. We describe the resulting decomposition of the VHS, and we give some results about the monodromy of the summands.

(4.1) Let (m, N, a) be as in 2.1. For n ∈ ℤ/mℤ, let N(n) be the number of indices i ∈ {1, …, N} such that na_{i} ≢ 0 modulo m. In particular, N(n) = N if n ∈ (ℤ/mℤ)^{∗}. If n ≠ 0 then d_{n} + d_{−n} = N(n) − 2. Let m′ ⩾ 2 be a divisor of m, say with m = rm′. Write N′ := N(r mod m), and consider the N′-tuple a′ in (ℤ/m′ℤ)^{N′} obtained from the N-tuple (a_{1} mod m′, …, a_{N} mod m′) by omitting the zero entries. Then (m′, N′, a′) is again a triple as in 2.1; we refer to it as the triple obtained from (m, N, a) by reduction modulo m′.

(4.2) As in 2.2, consider the family of curves f: C → T associated with the triple (m, N, a), where, as in the previous section, we work over ℂ. The cohomology groups H^{1}(C_{t}, ℚ) are the fibres of a polarized ℚ-VHS with underlying local system 𝕍 := R^{1}f_{∗}^{an}ℚ_{C}. This ℚ-VHS comes equipped with an action of the group ring ℚ[μ_{m}] = ∏_{d|m} K_{d}, where K_{d} = ℚ[t]/Φ_{d} is the cyclotomic field of dth roots of unity. Accordingly we have a decomposition of ℚ-VHS,

(4.2.1)  𝕍 = ⊕_{d|m}𝕍^{[d]} ,

where 𝕍^{[d]} is a polarized ℚ-VHS over T equipped with an action of K_{d}.

(4.3) Let (m, N, a) be as in 2.1, with N ⩾ 4. Let m′ ⩾ 2 be a proper divisor of m, and let (m′, N′, a′) be the triple obtained from (m, N, a) by reduction modulo m′, as in 4.1. Let ρ: 𝔸^{N} → 𝔸^{N′} be the projection map, omitting the coordinates x_{i} for all indices i with a_{i} ≡ 0 modulo m′. Performing the construction of 2.2 we may choose T = T (m, N, a) ⊂ 𝔸^{N} and T′ = T (m′, N′, a′) ⊂ 𝔸^{N′} such that ρ maps T to T′. (The map T → T′ is then dominant.) If f′: C′ → T′ is the family of curves associated with the triple (m′, N′, a′), the pull-back ρ^{∗}C′ = C′×_{T′} T can be identified with the quotient of C/T modulo the action of μ_{r} ⊂ μ_{m}, where r = m/m′. Let 𝕍′ = 𝕍(m′, N′, a′) be the ℚ-VHS over T′ associated with the triple (m′, N′, a′). The quotient morphism C → ρ^{∗}C′ over T gives a map ρ^{∗}𝕍′ → 𝕍 and this induces an isomorphism ρ^{∗}𝕍′ ≅ ⊕_{d|m′}𝕍^{[d]}. We refer to the sub-ℚ-VHS ρ^{∗}𝕍′ as the old part associated with the divisor m′. The direct summand 𝕍^{[m]} ⊂ 𝕍 is called the new part; in the decomposition (4.2.1) it is the only direct summand that is not contained in an old part.

(4.4) Lemma. — Let the notation and assumptions be as in 4.3. Let g = g(m, N, a) and g′ = g(m′, N′, a′) be the respective genera. If Z(m, N, a) ⊂ 𝒜_{g,ℂ} is a special subvariety then Z(m′, N′, a′) ⊂ 𝒜_{g′,ℂ} is special, too.

Proof. Suppose Z(m, N, a) is special. Write J = J(m, N, a) → T and J′ = J(m′, N′, a′) → T′ for the respective Jacobians. Then ρ^{∗}J′ is an isogeny factor of J. Hence, if θ: T → 𝒜_{g′,ℂ} is the morphism corresponding to ρ^{∗}J′, it follows from 1.4 that the Zariski closure of the image of θ is a special subvariety. But θ is just the composition of the projection ρ: T → T′, which is dominant, and the morphism ϕ(m′, N′, a′): T′ → 𝒜_{g′,ℂ} associated with the data (m′, N′, a′). Hence the closure of the image of θ is Z(m′, N′, a′). □

(4.5) With notation as in 2.5, the ℂ-local system 𝕍_{ℂ} decomposes as 𝕍_{ℂ} = ⊕_{n∈ℤ/mℤ}𝕍_{ℂ,(n)}. The relation with (4.2.1) is that 𝕍_{ℂ}^{[d]} is the sum of all 𝕍_{ℂ,(n)} with ord(n) = d. We have V_{ℂ,(0)} = 0, and for n ≠ 0 the ℂ-local system 𝕍_{ℂ,(n)} has rank N(n) − 2, where N(n) is defined as in 4.1. (See also [7], Section 2.) With real coefficients, and with I(m) as in (3.2.1), we have a decomposition of ℝ-VHS

𝕍_{ℝ} = ( ⨁_{±n∈I(m),2n≢0} 𝕍_{ℝ,(±n)}) ⊕ 𝕍_{ℝ,( \frac{m}{2} )} ,

where the last summand only occurs if m is even. The decomposition is such that

𝕍_{ℝ,(±n)} ⊗_{ℝ} ℂ = 𝕍_{ℂ,(n)} ⊕ 𝕍_{ℂ,(−n)}

and

𝕍_{ℝ,( \frac{m}{2} )} ⊗_{ℝ} ℂ = 𝕍_{ℂ,( \frac{m}{2} )} (for even m) .

On the summand 𝕍_{ℝ,(±n)} the polarization of the VHS induces a (−1)-hermitian form β_{±n} of signature (d_{n}, d_{−n}); see also [7], Corollary (2.21) and (2.23). In case m is even, the polarization induces a symplectic form β\frac{m}{2} on 𝕍_{ℝ,( \frac{m}{2} )}. As in 3.3 we choose a base point b ∈ T (ℂ), and a symplectic similitude σ: H^{1}(J_{b}, ℚ) ⟶^{∼} V. Via this similitude, we identify V with the fibre of 𝕍 at b. Correspondingly, we write V_{ℝ,(±n)} for the direct summand of V_{ℝ} that under σ maps to the fibre at b of 𝕍_{ℝ,(±n)}. For what follows, it will be convenient to choose the base point b to be a Hodgegeneric point with respect to the variation 𝕍, so from now on we assume this. Define Mon ⊂ GL(V) and Hdg ⊂ GL(V) to be the algebraic monodromy groups of the ℚ-local system 𝕍 and the Hodge group of the fibre at b, respectively. (Since we have chosen b to be Hodge-generic, Hdg is the generic Hodge group of the VHS.) By [3], Theorem 1, the identity component Mon^{0} is a semisimple normal subgroup of Hdg.

(4.6) Remark. — The generic Hodge group Hdg is contained in the algebraic group H := GL_{ℚ[μ_{m}]}(V) ∩ CSp(V, Ψ) considered in 3.3. Extending scalars to ℝ we have

(4.6.1)  H_{ℝ} ≅ (  ∏_{±n∈I(m),2n≢0} U(V_{ℝ,(±n)}, β_{±n})) × Sp(V_{ℝ,( \frac{m}{2} )}, β\frac{m}{2} ) ,

where the symplectic factor only occurs if m is even. Our formula (3.3.1) follows from this together with what was explained in 1.5.

We have Mon^{0} ⊂ Hdg ⊂ H. The next proposition gives us information about the projections of the ℝ-group Mon_{ℝ}^{0} onto the various factors in the decomposition (4.6.1).

(4.7) Proposition. — (i) Suppose we have ±n ∈ I(m) with 2n ≢ 0 such that d_{n} ⩾ 1 and d_{−n} ⩾ 1. Then the image of Mon_{ℝ}^{0} in U(V_{ℝ,(±n)}, β_{±n}) is the special unitary group SU(V_{ℝ,(±n)}, β_{±n}), which is isomorphic to SU(d_{n}, d_{−n}). (ii) If m is even, the group Mon_{ℝ}^{0} projects surjectively to the symplectic group Sp(V_{ℝ,( \frac{m}{2} )}, β\frac{m}{2} ), which is isomorphic to Sp_{2h,ℝ} with h = d\frac{m}{2} .

Proof. For (i), see Rohde [22], Theorem 5.1.1. For (ii), assume m is even, and let h := d\frac{m}{2} . There are then 2h+2 indices i for which a_{i} is even, and without loss of generality we may assume these are the indices i ∈ {1, …, 2h+2}. With the above notation, 𝕍_{ℝ,( \frac{m}{2} )} is the same as 𝕍_{ℝ}^{[2]}, and if we define Mon^{[2]} ⊂ GL(V^{[2]}) as the algebraic monodromy group of the ℚ-local system 𝕍^{[2]} then the image of Mon^{0} in Sp(V_{ℝ,( \frac{m}{2} )}, β\frac{m}{2} ) equals Mon_{ℝ}^{[2],0}. Further, 𝕍^{[2]} is just the local system obtained from the family of hyperelliptic curves u^{2} = (x − t_{1}) ⋯ (x − t_{2h+2}); cf. 4.3. By [1], Theorem 1 (see also [2]), the connected algebraic monodromy group of this local system is the full symplectic group. □

(4.8) Let (m, N, a) be a triple as in 2.1, with N ⩾ 4. Consider the following condition.

(4.8.1)

There exists a ±n ∈ I(m) with {d_{n}, d_{−n}} = {1, N − 3},
and for all other ±n′ ∈ I(m), either d_{n′} = 0 or d_{−n′} = 0.

All examples in Table 1, with the exception of Example 2, satisfy this condition. It has been proven by Rohde in [22] that these nineteen examples are the only examples (up to equivalence) of triples (m, N, a) with N ⩾ 4 that satisfy (4.8.1).

(4.9) We now prove Theorem 3.6 for m = 2. In this case N is even, g = (N − 2)/2, and up to equivalence a = (1, …, 1) is the only possibility. Further, Z(2, N, a) ⊂ 𝒜_{g,ℂ} is the closure of the hyperelliptic locus in 𝒜_{g}. The assertion is that this is not special for g ⩾ 3. This is true because for g ⩾ 3 the hyperelliptic locus is not dense in 𝒜_{g}, whereas by (ii) of Proposition 4.7 the generic Hodge group Hdg is the full symplectic group.

In what follows we may therefore assume m > 2.

5. Canonical liftings of ordinary Jacobians in positive characteristic

(5.1) We need to recall some definitions and results from the paper [8] by Dwork and Ogus. We first discuss their theory in a general setting. After that, from 5.4 on, we return to the specific families of curves that interest us, and we explain how the results of [8] apply to that situation. Given an irreducible base scheme T and a smooth projective curve f: C → T we write ω_{C/T} for Ω_{C/T}^{1} . Consider the Hodge bundle 𝔼 = 𝔼(C/T) := f_{∗}ω_{C/T}, which is a vector bundle on T of rank g, the genus of the fibres. We have a Kodaira-Spencer map KS: Sym^{2}(𝔼) → Ω_{T/R}^{1} and we define

𝒦 = 𝒦 (C/T) := Ker(Sym^{2}(𝔼) ⟶^{mult} f_{∗}(ω_{C/T}^{⊗2} )) ,

𝒬 = 𝒬(C/T) := Coker(f_{∗}(ω_{C/T}^{⊗2} )^{∨} ⟶^{mult^{∨}} Sym^{2}(𝔼)^{∨}) ,

where “mult” is the multiplication map. We remark that 𝒬 can be thought of as the pull-back to T of the normal bundle to the Torelli locus inside 𝒜_{g}. Outside the hyperelliptic locus, mult is surjective and 𝒦 is the dual of 𝒬.

(5.2) Let k be an algebraically closed field of characteristic p > 0. Consider a smooth projective curve C/k of genus g such that its Jacobian J is ordinary. Let λ be the natural principal polarization of J. By Serre-Tate theory (see [9] or [10], Chap. 5) we have a canonical lifting of (J, λ) to a principally polarized abelian variety (J^{can}, λ^{can}) over the ring of Witt-vectors W (k). In general, for g ⩾ 4 this canonical lifting is not the Jacobian of a curve over W (k). More precisely, Dwork and Ogus show that in general not even the canonical lifting over W_{2}(k), the Witt-vectors of length 2, is again a Jacobian. Following [8] we call the ordinary curve C/k pre-W_{2}-canonical if there exists a smooth projective curve Y over W_{2}(k) such that (J^{can}, λ^{can}) over W_{2}(k) is isomorphic, as a principally polarized abelian variety, to the Jacobian of Y. The main advantage of working modulo p^{2} is that in this case there is a natural class β_{C/k} in 𝒬(C/k) that vanishes if and only if C/k is pre-W_{2}-canonical; see [8], Prop. (2.4). This invariant β_{C/k} cannot be defined in a family of curves (in a sense that can be made precise, see [8], § 3), but its Frobenius pullback can, as we shall discuss next. Let T be a smooth k-scheme, and let F_{T}: T → T be its absolute Frobenius endomorphism. Consider a smooth projective curve f: C → T such that all fibres C_{t} are ordinary. This assumption permits us to view the inverse Cartier operator as an 𝒪_{T} -linear homomorphism

γ: F_{T}^{∗}𝔼 → 𝔼 ,

with 𝔼 = 𝔼(C/T) the Hodge bundle. This map γ is the inverse transpose of the Frobenius action on R^{1}f_{∗}𝒪_{C}. Dwork and Ogus define a global section β̃_{C/T} of F_{T}^{∗}𝒬(C/T) such that for any t ∈ T (k) the value of β̃_{C/T} at t is F_{k}^{∗}(β_{C_{t}/k}). See [8], Prop. 2.7. The sheaf F_{T}^{∗}𝒬(C/T) comes equipped with a canonical flat connection

∇: F_{T}^{∗}𝒬(C/T) → F_{T}^{∗}𝒬(C/T) ⊗ Ω_{T/k}^{1} .

In particular, still with T smooth over k, if C_{t}/k is pre-W_{2}-canonical for all t ∈ T (k), the section β̃_{C/T} is zero, so certainly ∇β̃_{C/T} = 0. One of the main points of [8], then, is that ∇β̃_{C/T} can be calculated explicitly. The result is easiest to state under the additional hypothesis that the fibres C_{t} are not hyperelliptic. We note that this assumption is not always satisfied in the situation we want to consider. In Prop. 5.8 we shall give a modified version of the next result that also works for hyperelliptic families. If the C_{t} are not hyperelliptic, the multiplication map Sym^{2}(𝔼) → f_{∗}(ω^{⊗2}) is surjective, so, with notation as introduced in 5.1, β̃_{C/T} can be viewed as an 𝒪_{T} -linear map F_{T}^{∗}𝒦 → 𝒪_{T}. The following result is [8], Prop. 3.2. (In [8] the result is stated under some versality assumption on the family C/T but one easily verifies that their Prop. 3.2 is true without this assumption.)

(5.3) Proposition (Dwork-Ogus). — With assumptions and notation as above, and assuming the curves C_{t} to be non-hyperelliptic, −∇(β̃_{C/T}): F_{T}^{∗}𝒦 → Ω_{T/k}^{1} is equal to the composition

F_{T}^{∗}𝒦 ⟶^{incl} F_{T}^{∗}Sym^{2}(𝔼) ⟶^{Sym^{2}(γ)} Sym^{2}(𝔼) ⟶^{KS} Ω_{T/k}^{1} .

(5.4) Assumptions. — In the rest of this section we fix data (m, N, a) as in 2.1 with m > 2 and N ⩾ 4, and we consider the family of curves f: C → T as constructed in 2.2. We assume that (m, N, a) is not equivalent to one of the triples listed in Table 1. We further assume that the closed subvariety Z(m, N, a) ⊂ 𝒜_{g,ℂ} is a special subvariety; our aim is to derive a contradiction. Without loss of generality we may suppose (m, N, a) is minimal with respect to the above assumptions, by which we mean that there is no triple (m′, N′, a′) that satisfies our assumptions and for which we have 4 ⩽ N′ < N or N′ = N and 2 ⩽ m′ < m. In particular, it follows from Lemma 4.4 that if m′ is a proper divisor of m and (m′, N′, a′) is obtained from (m, N, a) by reduction modulo m′, as in 4.3, either N′ ⩽ 3 or (m′, N′, a′) is equivalent to one of the twenty triples listed in Table 1.

(5.5) Lemma. — (i) The subsheaf f_{∗}(ω_{C/T}^{⊗2} )_{(0)} of μ_{m}-invariant sections of f_{∗}(ω_{C/T}^{⊗2} ) is a locally free 𝒪_{T} -module of rank N − 3. (ii) The Kodaira-Spencer morphism KS: Sym^{2}(𝔼) → Ω_{T/R}^{1} factors over the composite map

Sym^{2}(𝔼) ⟶^{pr} Sym^{2}(𝔼)_{(0)} ⟶^{mult} f_{∗}(ω_{C/T}^{⊗2} )_{(0)}

and the induced map KS_{(0)}: f_{∗}(ω_{C/T}^{⊗2} )_{(0)} → Ω_{T/R}^{1} is a fibrewise injective homomorphism of vector bundles.

Proof. (i) By a result of Chevalley and Weil in [5], for every t ∈ T (ℂ) the subspace of μ_{m}-invariants in H^{0}(C_{t}, Ω^{⊗2}) has dimension N − 3. (ii) The fibre of Sym^{2}(𝔼)_{(0)} at a point t is dual to the space H^{1}(C_{t}, Θ)_{(0)}, which parametrizes the deformations of C_{t} for which the μ_{m}-action deforms along. Since we have a μ_{m}-action on the whole family C/T, the Kodaira-Spencer map factors through the projection Sym^{2}(𝔼) → Sym^{2}(𝔼)_{(0)}. For any family of curves it also factors through the multiplication map Sym^{2}(𝔼) → f_{∗}(ω_{C/S}^{⊗2}); hence KS factors through f_{∗}(ω_{C/T}^{⊗2} )_{(0)}. The last assertion of (ii) just says that at any t ∈ T (k) our family is complete in the sense of deformation theory: if D → Spec(k[ϵ]) is a first-order deformation of C_{t} with its μ_{m}-action, D/k[ϵ] can be obtained by pull-back from our family. This is clear, as the quotient D/μ_{m} is isomorphic to ℙ_{k[ϵ]}^{1}. □

(5.6) Lemma. — There exists a prime number p with p ≡ 1 modulo m and, for 𝔭 a prime of R = ℤ[1/m, u]/Φ_{m} above p, a dense open subset U of T_{0} := T ⊗_{R} (R/𝔭), such that for any algebraically closed field k of characteristic p and any t ∈ U(k) the curve C_{t} is ordinary and the Jacobian J_{t} is pre-W_{2}-canonical.

Proof. Let p be any prime number with p ≡ 1 mod m. Note that in this case R/𝔭 = 𝔽_{p} for any prime 𝔭 ⊂ R above p. We write T ⊗ 𝔽_{p} for T ⊗_{R} (R/𝔭). By [4], Prop. 7.4, there is a Zariski open U ⊂ T ⊗ 𝔽_{p} such that C_{t} is ordinary for all t ∈ U(k). On the other hand, by a theorem of Noot [18], [19], in the slightly more precise formulation of [13], Thm. 4.2, the assumption that Z = Z(m, N, a) is special implies that for p large enough and t ∈ T_{0}(k) any ordinary point, the canonical lifting of J_{t} gives a W (k)-valued point of Z. In particular, J_{t} is then pre-W_{2}-canonical. □

(5.7) For most choices of (m, N, a) the general member in our family of curves C → T is not hyperelliptic. In this case we may, and will, choose p and U in Lemma 5.6 such that the curves C_{t} with t ∈ U(k) are non-hyperelliptic. It may happen, however, that Z(m, N, a) is fully contained in the hyperelliptic locus, in which case we call (m, N, a) a hyperelliptic triple. This occurs, for instance, if m = 2m′ is even and a is of the form (a_{1}, a_{2}, m′, …, m′), or if N = 4 and a = (1, 1, m − 1, m − 1). In such a case we will just choose p and U as in Lemma 5.6.

The following variant of Prop. 5.3 is the essential tool in our proof of Theorem 3.6. Let p and U be chosen as above, and let C_{U} → U be the restriction of C/T to U. With notation as in 2.5 and 5.1 we have eigenspace decompositions 𝒬(C_{U}/U) = ⊕_{n∈ℤ/mℤ} 𝒬_{(n)}; likewise for other 𝒪_{U}[μ_{m}]-modules such as 𝒦 (C_{U}/U) and 𝔼_{U} = 𝔼(C_{U}/U). Accordingly we can write β̃_{C_{U} /U} = ∑_{n} β̃_{(n)}, with β̃_{(n)} a section of F_{U}^{∗}𝒬_{(n)}.

(5.8) Proposition. — With assumptions and notation as in 5.4 and 5.7, the composite map (5.8.1)

F_{U}^{∗}𝒦_{(0)} ⟶^{incl} F_{U}^{∗}Sym^{2}(𝔼_{U})_{(0)} ⟶^{Sym^{2}(γ)} Sym^{2}(𝔼_{U})_{(0)} ⟶^{mult_{(0)}} f_{∗}(ω_{C/U}^{⊗2})_{(0)}

is zero.

Proof. If (m, N, a) is non-hyperelliptic this is immediate from Prop. 5.3 together with (ii) of Lemma 5.5. So from now on we may assume that Z(m, N, a) is contained in the hyperelliptic locus. Let ι ∈ Aut(C_{U}/U) be the hyperelliptic involution. We remark that ι may or may not be contained in the subgroup μ_{m} ⊂ Aut(C_{U}/U); by inspection of the hyperelliptic examples mentioned in 5.7 we see that both cases occur. Let ω = Ω_{C_{U} /U}^{1}. In the hyperelliptic case the multiplication map Sym^{2}(𝔼_{U}) → f_{∗}(ω^{⊗2}) is no longer surjective. However, if we denote the invariants under the action of ι by a subscript “even”,

mult_{(even)}: Sym^{2}(𝔼_{U})_{(even)} → f_{∗}(ω^{⊗2})_{(even)}

is again surjective; see for instance [21], Lemmas 2.12 and 2.13. The assumption that Z(m, N, a) is fully contained in the hyperelliptic locus implies that f_{∗}(ω^{⊗2})_{(0)} ⊂ f_{∗}(ω^{⊗2})_{(even)}. Hence also mult_{(0)} is surjective, and we may view β̃_{(0)} as an 𝒪_{U}-linear map F_{U}^{∗}𝒦_{(0)} → 𝒪_{U}. Our assumptions imply that β̃_{(0)}, hence also ∇β̃_{(0)}, is zero, so all that remains to be checked is that for the 0-component the analogue of Prop. 5.3 holds. For this we can follow the proof of [8], Prop. 3.2: The proof that loc. cit. diagram (3.2.9) is commutative does not use the assumption that the curves are non-hyperelliptic and therefore goes through in the hyperelliptic case. Moreover, in the notation of that proof, we can choose the lifted family Y /T (where Y/T is our C_{U}/U) such that it has an action of μ_{m}. Taking 0-components in diagram (3.2.9) then gives that −∇β̃_{(0)} equals the composition of (5.8.1) and KS_{(0)}, and by (ii) of Lemma 5.5 we conclude that the map (5.8.1) is zero. □

(5.9) Lemma. — Write q = (p − 1)/m. Let n ∈ ℤ/mℤ, and let A = A_{n}(t) ∈ GL_{d_{n}}(𝒪_{U}) be the matrix of the Cartier operator γ^{−1}: 𝔼_{U,(n)} → F_{U}^{∗}𝔼_{U,(n)} with regard to the frames

ω_{n,0}, …, ω_{n,d_{n}−1} , respectively ω_{n,0} ⊗ 1, …, ω_{n,d_{n}−1} ⊗ 1 .

Then the matrix coefficient A_{ρ,σ}, for ρ, σ ∈ {0, …, d_{n} − 1}, equals

(5.9.1) (−1)^{Σ} · ∑_{j_{1}+···+j_{N} =Σ} \binom{q · [−na_{1}]_{m}}{j_{1}} ⋯ \binom{q · [−na_{N}]_{m}}{j_{N}} · t_{1}^{j_{1}} ⋯ t_{N}^{j_{N}} ,

where Σ = (d_{n} − σ)(p − 1) + (ρ − σ).

Proof. This is the dual version of [4], Lemma 5.1, part (i). □

We remark that the result of Bouw in [4] is valid in a more general context. She uses this to prove the existence of cyclic covers whose p-rank reaches a natural upper bound imposed by the local monodromy data; see [4], Thm. 6.1.

6. Proof of the main result: four branch points

In this section we prove Theorem 3.6 in the case N = 4. Let us first try to explain the idea of the argument, by way of guide for the reader. Assume we have a triple (m, N, a) that is not one of the twenty examples in Table 1 but such that Z(m, N, a) is special. We want to show that this is impossible. We have the family of curves C → T over a ring R of finite type over ℤ; we then consider the reduction modulo p, with p chosen as in 5.7. The nontrivial information we have, exploiting the assumption that Z(m, N, a) is special, is that this gives a family of ordinary Jacobians in characteristic p such that the canonical liftings over W_{2}(k) are again Jacobians. In Prop. 5.8 we have translated this, based on the theory in [8], into the vanishing of a certain abstractly defined map. Lemma 5.9 enables us to calculate this map explicitly. To apply this, we need a differential form that gives us an element in the source of the map (5.8.1); we write down such a form in 6.2 below. The fact that the image of this element is zero then gives us a polynomial identity (6.2.1). By some combinatorial arguments we then show that this identity cannot hold if (m, N, a) is not equivalent to one of the twenty special triples in Table 1.

(6.1) We retain the assumptions made in 5.4. In addition we assume N = 4. Let

𝒟(m, a) := {±n ∈ I(m) | d_{n} = d_{−n} = 1} .

(Here I(m) is as defined in (3.2.1) and d_{n} is given by (3.2.2). Though not indicated in the notation, d_{n} depends on a.) If ±n ∈ 𝒟(m, a), the equalities d_{n} = d_{−n} = 1 imply that na_{i} ≢ 0 modulo m for all i ∈ {1, …, 4}. Hence,

(6.1.1) [−na_{i}]_{m} = m − [na_{i}]_{m} for all i,  and  ∑_{i=1}^{4} [−na_{i}]_{m} = 2m ,

where we recall that [b]_{m} denotes the representative of the class (b mod m) in {0, 1, …, m − 1}.

We first observe that dim S(μ_{m}) > 1. Indeed, if dim S(μ_{m}) = 1 then (4.8.1) holds, and by the results of [22], Chapter 6, (m, N, a) is one of the triples in Table 1, which contradicts our assumptions. For n ∈ ℤ/mℤ we have d_{n} + d_{−n} ⩽ 2, and if m is even then d\frac{m}{2} ⩽ 1. The inequality dim S(μ_{m}) > 1 is therefore equivalent to the fact that #𝒟(m, a) ⩾ 2. Choose two distinct pairs ±n and ±n′ in 𝒟(m, a). Note that if m is even, we may have n = −n or n′ = −n′.

(6.2) Choose p and U as in Lemma 5.6, where we may further assume that the curves C_{t} for t ∈ U(k) are either all non-hyperelliptic or all hyperelliptic. We keep the notation introduced in the previous section; in particular we recall that 𝒦 (C_{U}/U) = ⊕_{n∈ℤ/mℤ} 𝒦_{(n)}. It follows from (6.1.1) and the definition of the forms ω_{n,ν} in (2.6.1) that

η := ω_{n,0} ⊗ ω_{−n,0} − ω_{n′,0} ⊗ ω_{−n′,0}

is a section of 𝒦_{(0)}. For ν ∈ {±n, ±n′} the matrix A_{ν} = A_{ν}(t) of Lemma 5.9 is a polynomial in 𝔽_{p}[t_{1}, …, t_{4}]. As the Jacobians of the curves C_{t} for t ∈ U(k) are ordinary, the Cartier operator γ^{−1} in Lemma 5.9 is an isomorphism of vector bundles, so A_{ν}(t) is invertible as a section of 𝒪_{U}. Because ω_{n,0} · ω_{−n,0} = ω_{n′,0} · ω_{−n′,0} is a non-zero section of f_{∗}(ω^{⊗2}), it follows from Prop. 5.8 that

(6.2.1)  A_{n} · A_{−n} = A_{n′} · A_{−n′}

as polynomials. Define B_{ν} := A_{ν}|_{t_{1}=0}, the polynomial obtained from A_{ν} by substituting t_{1} = 0. Explicitly, (6.2.2)

B_{ν} = (−1)^{p−1} · ∑_{j_{2}+j_{3}+j_{4}=p−1} \binom{q · [−νa_{2}]_{m}}{j_{2}} ⋯ \binom{q · [−νa_{4}]_{m}}{j_{4}} · t_{2}^{j_{2}}t_{3}^{j_{3}}t_{4}^{j_{4}} .

Fix an index ι ∈ {2, 3, 4}, and write {1, 2, 3, 4} = {1, ι} ⨿ {κ, λ}. Let v_{n}(ι) be the largest integer v such that B_{n} is divisible by t_{ι}^{v}. From (6.2.2) and (6.1.1) we find

\begin{aligned}v_{n}(ι) & = max\{0, (p − 1) − q · [−na_{κ}]_{m} − q · [−na_{λ}]_{m}\} \\ & = max\{0, q · [na_{κ}]_{m} + q · [na_{λ}]_{m} − (p − 1)\} , \end{aligned}

and similarly for v_{−n}(ι). Next let w_{±n}(ι) be the largest integer w such that B_{n} · B_{−n} is divisible by t_{ι}^{w}. We find

\begin{aligned}w_{±n}(ι) & = max\{0, q · [na_{κ}]_{m} + q · [na_{λ}]_{m} − (p − 1)\} \\ & + max\{0, q · [−na_{κ}]_{m} + q · [−na_{λ}]_{m} − (p − 1)\} , \end{aligned}

which by (6.1.1) we can rewrite as

(6.2.3)

w_{±n}(ι) = max{0, q · [na_{κ}]_{m} + q · [na_{λ}]_{m} − (p − 1)}
+ max{0, q · [na_{1}]_{m} + q · [na_{ι}]_{m} − (p − 1)}

= q · max{[na_{1}]_{m} + [na_{ι}]_{m}, [na_{κ}]_{m} + [na_{λ}]_{m}} − (p − 1) .

(6.3) Lemma. — Consider two 4-tuples (b_{1}, b_{2}, b_{3}, b_{4}) and (b′_{1}, b′_{2}, b′_{3}, b′_{4}) in {1, …, m − 1}^{4} with ∑_{i=1}^{4} b_{i} = 2m = ∑_{i=1}^{4} b′_{i}. Suppose that for all partitions {1, 2, 3, 4} = {1, ι} ⨿ {κ, λ} we have {b_{1} + b_{ι}, b_{κ} + b_{λ}} = {b′_{1} + b′_{ι}, b′_{κ} + b′_{λ}} as sets. Then there is an even permutation σ ∈ A_{4} of order 2 such that either b′_{i} = b_{σ(i)} for all i or b′_{i} = m − b_{σ(i)} for all i.

Proof. Straightforward checking of all 8 = 2^{3} possible combinations (two for each partition). □

(6.4) The relation (6.2.1) implies that for any partition {1, 2, 3, 4} = {1, ι} ⨿ {κ, λ} we have w_{±n}(ι) = w_{±n′}(ι). Since ∑_{i=1}^{4} [na_{i}]_{m} = 2m = ∑_{i=1}^{4} [n′a_{i}]_{m}, it follows from (6.2.3) that

{[na_{1}]_{m} + [na_{ι}]_{m}, [na_{κ}]_{m} + [na_{λ}]_{m}} = {[n′a_{1}]_{m} + [n′a_{ι}]_{m}, [n′a_{κ}]_{m} + [n′a_{λ}]_{m}}

as sets. By the lemma it follows, possibly after replacing n by −n, which we may do, that there is an even permutation σ ∈ A_{4} of order 2 such that [n′a_{i}]_{m} = [na_{σ(i)}]_{m} for all i. As the pairs ±n and ±n′ are distinct, σ ≠ 1. Possibly after a permutation of the indices we may therefore assume that

(6.4.1)  \begin{matrix}[n′a_{1}]_{m} = [na_{2}]_{m} , & [n′a_{3}]_{m} = [na_{4}]_{m} , \\ [n′a_{2}]_{m} = [na_{1}]_{m} , & [n′a_{4}]_{m} = [na_{3}]_{m} . \end{matrix}

Since gcd(m, a_{1}, …, a_{4}) = 1 this implies that gcd(m, n) = gcd(m, n′).

(6.5) Let r = gcd(n, m). Let m′ = m/r, and consider the triple (m′, N′, a′) obtained from (m, N, a) by reduction modulo m′. Then we have a bijection 𝒟(m, a) ⟶^{∼} 𝒟(m′, a′) by (±n) ↦ (±\frac{n}{r}). Hence #𝒟(m′, a′) ⩾ 2. In particular, N′ = 4, and (m′, N′, a′) is not equivalent to one of the triples listed in Table 1. By our minimality assumption on (m, N, a), see 5.4, it therefore follows that r = 1. Hence we may replace the 4-tuple a by n′a = (n′a_{1}, …, n′a_{4}), which means we may set n′ = 1 in the above. In this case, (6.4.1) tells us that n^{2} = 1 in (ℤ/mℤ), and the 4-tuple a has the form a = (a_{1}, na_{1}, a_{3}, na_{3}) with n ≠ ±1 and

(6.5.1)  (n + 1)(a_{1} + a_{3}) = 2m .

From these relations we want to deduce that there is a pair ±ν ∈ 𝒟(m, a) with gcd(m, ν) ≠ 1, thereby obtaining a contradiction. Write m = 2^{k} · M with M odd. First assume n ≢ −1 modulo M. For an odd prime number ℓ and an exponent e ⩾ 1, the congruence n^{2} ≡ 1 mod ℓ^{e} only has n ≡ ±1 as solutions. Hence there is an odd prime divisor ℓ of m such that n ≢ −1 mod ℓ. But then (6.5.1) gives a_{1} + a_{3} ≡ 0 mod ℓ, and since gcd(m, a_{1}, …, a_{4}) = 1 we further have a_{1} ≢ 0 mod ℓ and a_{3} ≢ 0 mod ℓ. Taking ν := m/ℓ we find that ±ν ∈ 𝒟(m, a); contradiction.

The only remaining possibility is that n ≡ −1 modulo M, and therefore n ≢ −1 mod 2^{k}. In particular, k ⩾ 2. In this case (6.5.1) implies a_{1} +a_{3} ≡ 0 mod 4, in which case ±m/4 ∈ 𝒟(m, a); contradiction. This completes the proof of Thm. 3.6 in the case N = 4.

7. Proof of the main result: five or more branch points

We now turn to the case N ⩾ 5. The idea behind the proof is the same as in the previous section, but there are some additional difficulties. In 7.2 and 7.3 we prove two technical results. Once we have these, we can again write down elements in the source of the map (5.8.1). Similar to what we did in Section 6, this gives us polynomial identities (7.4.1), and with some elementary combinatorial arguments we can then conclude the proof.

(7.1) Assumptions. — We retain the assumptions made in 5.4. In addition we now assume N ⩾ 5. Again we assume (m, N, a) is minimal. Because we have already proven Thm. 3.6 for N = 4, it is again true that if m′ is a proper divisor of m and (m′, N′, a′) is obtained from (m, N, a) by reduction modulo m′, either N′ ⩽ 3 or (m′, N′, a′) is equivalent to one of the twenty triples listed in Table 1.

(7.2) Lemma. — With assumptions as in 7.1, there exists an index n ∈ (ℤ/mℤ)^{∗} such that {d_{n}, d_{−n}} ≠ {0, N − 2}.

With the terminology introduced in 4.3, the lemma says that the new part of the local system 𝕍 (as in Section 4) is not isotrivial.

Proof. We assume {d_{n}, d_{−n}} = {0, N − 2} for all n ∈ (ℤ/mℤ)^{∗}, and we seek to derive a contradiction. It will be convenient to consider the function δ_{a}: ℤ/mℤ → ℤ_{⩾0} given by δ_{a}(n) = ∑_{i=1}^{N} ⟨\frac{−na_{i}}{m} ⟩. Note that for n ≠ 0 the d_{n} defined in (3.2.2) equals −1 + δ_{a}(n). For n ∈ (ℤ/mℤ)^{∗} we have δ_{a}(n) ⩾ 1 and δ_{a}(n) + δ_{a}(−n) = N. Our assumption is equivalent to the condition that

(7.2.1)  {δ_{a}(n), δ_{a}(−n)} = {1, N − 1} for all n ∈ (ℤ/mℤ)^{∗}.

Possibly after replacing a = (a_{1}, …, a_{N}) by −a, we may assume δ_{a}(1) = 1, which means that

(7.2.2)  [a_{1}]_{m} + ⋯ + [a_{N}]_{m} = m .

We divide the proof into a couple of steps.

Step 1. Our first goal is to show that m > 60. For a given m, (7.2.2) leaves only finitely many possible triples (m, N, a), up to equivalence. With the help of a small computer program we check that for m ⩽ 60 the only two possibilities with N ⩾ 5 and {δ_{a}(n), δ_{a}(−n)} = {1, N − 1} for all n ∈ (ℤ/mℤ)^{∗} are given (up to equivalence) by

(7.2.3)  m = 6 , N = 5 , a = (1, 1, 1, 1, 2) ;

(7.2.4)  m = 6 , N = 6 , a = (1, 1, 1, 1, 1, 1) .

So it remains to show that for these triples, Z(m, N, a) is not special. In case (7.2.3), we have g = 7 and the family of Jacobians J → T decomposes, up to isogeny, as a product of three factors: J ∼ Y_{1} × Y_{2} × Y_{3}, with

— Y_{1} a pull-back of the Legendre family of elliptic curves, which is Example 1 in Table 1; this is the old factor corresponding to the divisor 2 of m,

— Y_{2} a pull-back of Example 6; this is the old factor corresponding to the divisor 3 of m,

— Y_{3} the new part. (The new part Y_{3} is in fact an isotrivial family of 3-dimensional abelian varieties of CM type, but we will not need this.) Let b ∈ T (ℂ) be a Hodge-generic point. The fibre Y_{1,b} is an elliptic curve with endomorphism ring ℤ; its Hodge group is isomorphic with SL_{2}. The fibre Y_{2,b} is an abelian threefold that is easily seen to be simple. (E.g., use the fact that Example 6 is special, together with [23], Thm. 5.) By [16], Prop. 3.8 the Hodge group of Y_{1,b} ×Y_{2,b} is the product of the two Hodge groups. Hence the smallest special subvariety S ⊂ 𝒜_{7} that contains Z(m, N, a) has dimension at least 1 + 2 = 3 (see 1.5), and since N − 3 = 2 < 3 this means that Z(m, N, a) is not special. In case (7.2.4), we have g = 10 and the family of Jacobians J → T decomposes, up to isogeny, as a product of three factors: J ∼ Y_{1} × Y_{2} × Y_{3}, with

— Y_{1} a pull-back of Example 2; this is the old factor corresponding to the divisor 2 of m,

— Y_{2} a pull-back of Example 10; this is the old factor corresponding to the divisor 3 of m,

— Y_{3} the new part. (In this case the new part Y_{3} is an isotrivial family of 4-dimensional abelian varieties of CM type; again we will not need this.) For a Hodge-generic point b ∈ T (ℂ) the fibre Y_{1,b} is an abelian surface with endomorphism ring ℤ; its Hodge group H_{1} = Hdg(Y_{1,b}) is isomorphic with Sp_{4}. The fibre Y_{2,b} is an abelian fourfold. By Thm. 1.1 of [26] we have End(Y_{2,b}) = ℤ[ζ_{3}], and the tangent multiplicities for the factor Y_{2} (=Example 10) are given by d_{(1 mod 3)} = 3 and d_{(2 mod 3)} = 1. By [15], 7.4, it follows that for the generic Hodge group H_{2} = Hdg(Y_{2,b}) we have H_{2,ℝ} ≅ U(1, 3). If H = Hdg(J_{b}) is the generic Hodge group in our family of Jacobians, H projects surjectively to H_{1} and H_{2}. It follows that among the simple factors of H_{ℝ}^{ad} both a factor PSp_{4} and a factor PSU(1, 3) occur; this implies that the smallest special subvariety S ⊂ 𝒜_{7} that contains Z(m, N, a) has dimension at least 3 + 3 = 6 (again see 1.5), and since N − 3 = 3 < 6 this means that Z(m, N, a) is not special.

Step 2. The function δ_{a} has the property that δ_{a}(n_{1} + n_{2}) ⩽ δ_{a}(n_{1}) + δ_{a}(n_{2}). In particular, if n_{1}, n_{2} and n_{1} + n_{2} are all in (ℤ/mℤ)^{∗} then by (7.2.1) we have the implication

(7.2.5)  δ_{a}(n_{1}) = δ_{a}(n_{2}) = 1 ⇒ δ_{a}(n_{1} + n_{2}) = 1 .

Step 3. Next we show that m is not divisible by 2, 3 or 5. Let p ∈ {2, 3, 5}, and suppose m is divisible by p. Let m′ = m/p and let (m′, N′, a′) be obtained from (m, N, a) by reduction modulo m′. The new triple cannot be one of the examples listed in Table 1, for this would imply (by inspection of the table) that m = p · m′ ⩽ 5 · 12 = 60, which we have excluded. Hence our minimality assumption on the triple (m, N, a) implies that N′ ⩽ 3. Since N ⩾ 5, there are at least two indices i such that a_{i} ≡ 0 modulo m′. For p = 2 this contradicts (7.2.2). Hence 2 ∈ (ℤ/mℤ)^{∗} and by repeated application of (7.2.5), starting from δ_{a}(1) = 1, we find that δ_{a}(2^{k}) = 1 for all k. For p = 3 the only possibility left, in view of (7.2.2), is that there are precisely two indices i with [a_{i}]_{m} = m/3 and that [a_{i}]_{m} < m/3 for all remaining indices. This contradicts the fact that δ_{a}(2) = 1. Hence 3 does not divide m. Finally take p = 5. The fact that δ_{a}(2) = 1 implies that there is a unique index i with [a_{i}]_{m} > m/2 and that [a_{i}]_{m} < m/2 for all remaining indices. Since N ⩾ 5 and N′ ⩽ 3 this leaves (possibly after a permutation of the indices) two possibilities: either

[a_{1}]_{m} = 3m/5 , [a_{2}]_{m} = m/5 , [a_{i}]_{m} < m/5 for all i > 2,

or

m/2 < [a_{1}]_{m} < 3m/5 , [a_{2}]_{m} = [a_{3}]_{m} = m/5 , [a_{i}]_{m} < m/5 for all i > 3.

In both cases we get a contradiction with the fact that δ_{a}(4) = 1.

Step 4. We now show that m has at most four distinct prime divisors. To see this, let p be any prime divisor of m, let m′ = m/p and let (m′, N′, a′) be obtained from (m, N, a) by reduction modulo m′. If the new triple is one of the examples listed in Table 1, it can only be Example 17, for otherwise inspection of the table gives that m′ (and hence m) is divisible by 2, 3, or 5, which we have excluded. Example 17 has N′ = 4 branch points, so this fact, together with our minimality assumption on the triple (m, N, a), implies that N′ ⩽ 4. The previous argument shows that for any prime divisor p of m, there are at most 4 indices i such that a_{i} is not divisible by m/p. On the other hand, a_{i} ≢ 0 mod m, so for a given index i there can be at most one prime divisor p of m such that a_{i} is divisible by m/p. As N ⩾ 5 it follows that m has at most four distinct prime divisors.

Step 5; conclusion of the proof. Combining the conclusion of Step 3 with (7.2.5), we find that for all ρ ∈ {1, 2, 3, 4, 5} we have (ρ mod m) ∈ (ℤ/mℤ)^{∗} and δ_{a}(ρ mod m) = 1. Let ν be an integer such that gcd(m, ν) = 1 and δ_{a}(ν mod m) = 1. Let ρ be the smallest positive integer such that gcd(m, ν + ρ) = 1. The fact that m has at most four distinct prime divisors, all greater than 5, implies that ρ ⩽ 5. But then gcd(ρ, m) = 1 and from (7.2.5) we get that δ_{a}(ν + ρ mod m) = 1. Repeating this, we find that δ_{a}(n) = 1 for all n ∈ (ℤ/mℤ)^{∗}, which contradicts the fact that δ_{a}(n) + δ_{a}(−n) = N > 3. This completes the proof of the lemma. □

(7.3) Our next goal is to show there exist two distinct pairs ±n and ±n′ in I(m) with

(7.3.1)  d_{−n} = d_{−n′} = 1 , and d_{n} = d_{n′} = N − 3 .

(In particular, n ≢ 0 and n′ ≢ 0.) For this we work over ℂ and we consider the generic Mumford-Tate group M of the family C/T. As in 1.5, let M_{ℝ}^{ad} = Q_{1} × ⋯ × Q_{r} be the decomposition of M_{ℝ}^{ad} as a product of simple factors. The assumption that Z(m, N, a) is special means that N − 3 = ∑_{i=1}^{r} d(Q_{i}), with notation as in 1.5. Further, this assumption implies that the connected algebraic monodromy group Mon^{0} of the family is a normal subgroup of M; in particular, Mon_{ℝ}^{0,ad} = ∏_{i∈K} Q_{i} for some subset K ⊂ {1, …, r}. By Lemma 7.2 there exists an index n in (ℤ/mℤ)^{∗} with {d_{n}, d_{−n}} ≠ {0, N − 2}. By Prop. 4.7 it follows that one of the simple factors Q_{i}, say Q_{1}, is a PSU(d_{n}, d_{−n}), which gives d(Q_{1}) = d_{n}d_{−n}. Because n ∈ (ℤ/mℤ)^{∗} we have d_{n}+d_{−n} = N −2; hence d_{n}d_{−n} ⩾ N −3 with equality of and only if {d_{n}, d_{−n}} = {1, N − 3}. Possibly after replacing n by −n we therefore have d_{n} = N − 3 and d_{−n} = 1. Further, the relation N − 3 = ∑_{i=1}^{r} d(Q_{i}) implies that all other simple factors Q_{2}, …, Q_{r} are anisotropic (i.e., compact). Our assumptions imply that there is another index pair ±n′ ∈ I(m) with d_{n′} ≠ 0 and d_{−n′} ≠ 0, for if this is not the case then (m, N, a) satisfies (4.8.1), which implies it is one of the examples of Table 1. If m = 2m′ is even and n′ = m′ then by Prop. 4.7 one of the factors Q_{i} is a PSp_{2h} with h = d_{n′} > 0. As N − 3 ⩾ 2, this factor cannot be the factor Q_{1} = PSU(1, N − 3), and since PSp_{2h} is not anisotropic we arrive at a contradiction. So n′ ≠ −n′, and, again by Prop. 4.7, one of the Q_{i} is a non-compact unitary factor PSU(d_{n′}, d_{−n′}). This factor must be the factor Q_{1}; hence {d_{n′}, d_{−n′}} = {1, N − 3}, which gives what we want.

(7.4) Consider two distinct pairs ±n and ±n′ in I(m) for which (7.3.1) holds. Let Γ ∈ GL_{N−3}(𝒪_{U}) be the matrix of γ_{(n)}: F_{U}^{∗}𝔼_{U,(n)} → 𝔼_{U,(n)} with regard to the frames given by the forms ω_{n,σ}. It is the inverse of the matrix A = A_{n} of Lemma 5.9. Similarly, let Γ′ be the matrix of γ_{(n′)}, and let A′ = A_{n′} be its inverse. Further, let c and c′ be the sections of 𝒪_{U}^{∗} (the inverses of the 1 × 1 matrices A_{−n} and A_{−n′} of Lemma 5.9) such that γ(ω_{−n,0} ⊗ 1) = c · ω_{−n,0} and γ(ω_{−n′,0} ⊗ 1) = c · ω_{−n′,0}. For σ ∈ {0, 1, …, N − 4}, let η_{σ} ∈ Γ(U, 𝒦_{(0)}) be given by

η_{σ} := ω_{−n,0} ⊗ ω_{n,σ} − ω_{−n′,0} ⊗ ω_{n′,σ} .

As ω_{−n,0} · ω_{n,ρ} = ω_{−n′,0} · ω_{n′,ρ} as sections of f_{∗}(ω^{⊗2}), the image of η_{σ} under Sym^{2}(γ) equals

∑_{ρ=0}^{N−4} (c · Γ_{ρ,σ} − c′ · Γ′_{ρ,σ}) · (ω_{−n,0} · ω_{n,ρ}) .

Because the sections ω_{−n,0}·ω_{n,ρ} for ρ ∈ {0, …, N −4} are linearly independent, it follows from Prop. 5.8 that c · Γ_{ρ,σ} − c′ · Γ′_{ρ,σ} for all σ, ρ ∈ {0, 1, …, N − 4}, which can be rewritten as

(7.4.1)  c^{−1} · A_{ρ,σ} = c^{′,−1} · A′_{ρ,σ} .

Choose two distinct indices κ, λ in {1, 2, …, N}, and write {1, …, N} = {κ, λ} ⨿ I. For ν ∈ {0, 1}, define v_{n}(ν) to be the largest integer v such that A_{ν,ν}|_{t_{κ}=0} is divisible by t_{λ}^{v}. Similarly, let v_{−n} be the largest integer v such that c^{−1}|_{t_{κ}=0} is divisible by t_{λ}^{v}. Using the explicit formulas for c^{−1} and the matrix A from Lemma 5.9, we find

\begin{aligned}v_{−n} & = max\{0, (p − 1) − q · ∑_{i∈I} [na_{i}]_{m}\} , \\ v_{n}(0) & = max\{0, (N − 3)(p − 1) − q · ∑_{i∈I} [−na_{i}]_{m}\} , \\ v_{n}(1) & = 0 . \end{aligned}

(Recall that q = (p − 1)/m.) Next we define w_{n}(ν) = v_{−n} + v_{n}(ν) to be the largest integer w such that (c^{−1} · A_{ν,ν})|_{t_{κ}=0} is divisible by t_{λ}^{w}. Similar to (6.1.1), it follows from the fact that d_{−n} = 1 and d_{n} = N − 3 that [−na_{i}]_{m} = m − [na_{i}]_{m} for all i and ∑_{i=1}^{N} [na_{i}]_{m} = 2m. Using these relations we obtain

w(0) = max{(p − 1) − q · ∑_{i∉I} [na_{i}]_{m}, (p − 1) − q · ∑_{i∈I} [na_{i}]_{m}}

= |(p − 1) − q · ∑_{i∉I} [na_{i}]_{m}| = q · |m − [na_{κ}]_{m} − [na_{λ}]_{m}| ,

w(1) = max{0, (p − 1) − q · ∑_{i∈I} [na_{i}]_{m}}

= max{0, (p − 1) − q · ∑_{i∉I} [na_{i}]_{m}} = q · max{0, [na_{κ}]_{m} + [na_{λ}]_{m} − m} .

We can do the same calculations for the pair ±n′; let us call w′(0) and w′(1) the resulting values. From (7.4.1) we get the relations w(0) = w′(0) and w(1) = w′(1). It follows that for all choices of κ and λ we have [na_{κ}]_{m} + [na_{λ}]_{m} = [n′a_{κ}]_{m} + [n′a_{λ}]_{m}. This readily implies that [na_{i}]_{m} = [n′a_{i}]_{m} for all i ∈ {1, …, N}. As gcd(m, a_{1}, …, a_{N}) = 1 it follows that n = n′, which contradicts our assumption that the pairs ±n and ±n′ are distinct. This completes the proof of Theorem 3.6. □

References

[1] N. A’Campo, Tresses, monodromie et le groupe symplectique. Comment. Math. Helv. 54 (1979), 318–327.

[2] J. Achter, R. Pries, The integral monodromy of hyperelliptic and trielliptic curves. Math. Ann. 338 (2007), 187–206.

[3] Y. André, Mumford-Tate groups of mixed Hodge structures and the theorem of the fixed part. Compositio Math. 82 (1992), 1–24.

[4] I. Bouw, The p-rank of ramified covers of curves. Compositio Math. 126 (2001), 295–322.

[5] C. Chevalley, A. Weil, Über das Verhalten der Integrale 1. Gattung bei Automorphismen des Funktionenkörpers. Abh. Math. Semin. Univ. Hambg. 10 (1934), 358–361.

[6] J. de Jong, R. Noot, Jacobians with complex multiplication. In: Arithmetic algebraic geometry (Texel, 1989), Progr. Math. 89, Birkhäuser, 1991, pp. 177– 192.

[7] P. Deligne, G. Mostow, Monodromy of hypergeometric functions and nonlattice integral monodromy. Inst. Hautes Études Sci. Publ. Math. 63 (1986), 5–89.

[8] B. Dwork, A. Ogus, Canonical liftings of Jacobians. Compositio Math. 58 (1986), 111–131.

[9] N. Katz, Serre-Tate local moduli. In: Algebraic surfaces, Lecture Notes in Math. 868, Springer-Verlag, 1981, pp. 138–202.

[10] W. Messing, The crystals associated to Barsotti-Tate groups: with applications to abelian schemes. Lecture Notes in Math. 264, Springer-Verlag, 1972.

[11] B. Moonen, Special points and linearity properties of Shimura varieties. PhD thesis, University of Utrecht, 1995.

[12] B. Moonen, Linearity properties of Shimura varieties. I. J. Alg. Geom. 7 (1998), 539–567.

[13] B. Moonen, Linearity properties of Shimura varieties. II. Compositio Math. 114 (1998), 3–35.

[14] B. Moonen, F. Oort, The Torelli locus and special subvarieties. Preprint, August 2010, 43pp.

[15] B. Moonen, Yu. Zarhin, Hodge classes and Tate classes on simple abelian fourfolds. Duke Math. J. 77 (1995), 553–581.

[16] B. Moonen, Yu. Zarhin, Hodge classes on abelian varieties of low dimension. Math. Ann. 315 (1999), 711–733.

[17] G. Mostow, On discontinuous action of monodromy groups on the complex n-ball. J. Amer. Math. Soc. 1 (1988), 555–586.

[18] R. Noot, Hodge classes, Tate classes, and local moduli of abelian varieties. PhD thesis, University of Utrecht, 1992.

[19] R. Noot, Models of Shimura varieties in mixed characteristic. J. Alg. Geom. 5 (1996), 187–207.

[20] R. Noot, Correspondances de Hecke, action de Galois et la conjecture d’André-Oort (d’après Edixhoven et Yafaev). Sém. Bourbaki, Exp. 942.

Astérisque 307 (2006), 165–197.

[21] F. Oort, J. Steenbrink, The local Torelli problem for algebraic curves. In: Journées de Géometrie Algébrique d’Angers, Sijthoff & Noordhoff, 1980, pp. 157–204.

[22] J. Rohde, Cyclic coverings, Calabi-Yau manifolds and complex multiplication. Lecture Notes in Math. 1975, Springer, 2009.

[23] G. Shimura, On analytic families of polarized abelian varieties and automorphic functions. Ann. of Math. (2) 78 (1963), 149–192.

[24] G. Shimura, On purely transcendental fields of automorphic functions of several variable. Osaka J. Math. 1 (1964), 1–14.

[25] A. Yafaev, A conjecture of Yves André’s. Duke Math. J. 132 (2006), 393– 407.

[26] Yu. Zarhin, The endomorphism rings of Jacobians of cyclic covers of the projective line. Math. Proc. Cambridge Philos. Soc. 136 (2004), 257–267.

Ben Moonen

University of Amsterdam

KdV Institute for Mathematics PO Box 94248 1090 GE Amsterdam The Netherlands 820

Documenta Mathematica 15 (2010)