Annales Academiæ Scientiarum Fennicæ Series A. I. Mathematica Volumen 20, 1995, 251–257

A TRACE-CLASS RIGIDITY THEOREM FOR KLEINIAN GROUPS

Peter A. Perry

University of Kentucky, Department of Mathematics Lexington, Kentucky 40506–0027, U.S.A.; perry@ms.uky.edu

Abstract. Suppose that Γ_{1} and Γ_{2} are geometrically finite, convex co-compact, discrete groups of isometries of real hyperbolic space H^{3} whose domains of discontinuity are diffeomorphic. We show that if the respective scattering matrices S_{1}(s) and S_{2}(s) differ from each other by a traceclass perturbation on the unitary axis Re(s) = 1, then Γ_{1} and Γ_{2} are conjugate in PSL(2, C). This result reflects the rigidity of hyperbolic three-manifolds.

1. Introduction

In this note, we prove a rigidity theorem for the scattering operator associated to a co-infinite volume, convex co-compact discrete group of isometries of hyperbolic three-dimensional space H^{3} , modelled as the unit ball in R^{3} with geometric boundary S^{2} . The group of isometries of H^{3} may be realized as PSL(2, C). Let Γ be a discrete group of isometries of H^{3} , let Ω(Γ) denote the domain of discontinuity of Γ acting on S^{2} , and let Λ(Γ) denote the limit set of Γ. The group Γ is said to be geometrically finite if it admits a finite-sided fundamental domain, to have co-infinite volume if vol (H^{3}/Γ) is infinite, and to be convex co-compact if the fundamental domain does not touch the limit set of Γ. In what follows, we will always assume that Γ is geometrically finite, has co-infinite volume, and is convex co-compact. For such groups, it is known that M = H^{3}/Γ is a smooth Riemannian manifold whose geometric boundary (boundary at infinity) is a compact manifold B conformally equivalent to Ω(Γ)/Γ. The manifold B is a finite union of connected components each of which are compact manifolds without boundary equipped with a natural conformal structure.

Associated to such a manifold M is the scattering operator for the Laplacian Δ on M. The scattering operator S(s) for a complex parameter s is a pseudodifferential operator with a known singularity mapping smooth, Γ-automorphic forms of complex weight 2 − s on Ω(Γ) to smooth, Γ-automorphic forms of complex weight s. It connects incoming and outgoing generalized eigenfunctions for the Laplacian on M and its values along the axis Re(s) = 1 give the contribution of the continuous spectrum in the Selberg trace formula for Γ.

1991 Mathematics Subject Classification: Primary 58G25; Secondary 30F40.

Supported in part by NSF grant DMS-9343893.

It is natural to ask under what circumstances the scattering matrices for two hyperbolic manifolds M_{1} and M_{2} may be compared, i.e., under what circumstances it is possible to do ‘geometric perturbation theory’ of Laplacians. Intuitively two manifolds should have comparable ‘geometry at infinity’ for perturbation theory to be possible.

If M_{1} = H^{2}/Γ_{1} and M_{2} = H^{2}/Γ_{2} with Γ_{1} and Γ_{2} convex co-compact and co-infinite volume, the scattering matrices for M_{1} and M_{2} may be compared so long as the boundaries B_{1} and B_{2} are diffeomorphic. This simply means that M_{1} and M_{2} have the same number of ends since the ends of such hyperbolic surfaces are isometric to cylinders and the boundary components are all circles. In this case, the difference of scattering matrices, suitably defined, is a traceclass operator on an appropriate Hilbert space of functions on the boundary. One can then exploit the methods of trace-class scattering theory and the theory of Fredholm determinants on trace-class operators to produce a useful relative trace formula (see [9]).

It is natural to ask whether a similar procedure will work in three dimensions, i.e., whether two manifolds M_{1} = H^{3}/Γ_{1} and M_{2} = H^{3}/Γ_{2} with diffeomorphic boundaries B_{1} and B_{2} can be compared in the same way. In order to compare the scattering operators, it is necessary to assume also that the diffeomorphism ψ that maps B_{1} to B_{2} lifts to a diffeomorphism of the domains of discontinuity Ω(Γ_{1}) and Ω(Γ_{2}) that induces an isomorphism of the groups Γ_{1} and Γ_{2} . This is essentially equivalent to requiring that ψ induce an invertible map from Γ_{1} - automorphic forms on Ω(Γ_{1}) to Γ_{2} -automorphic forms on Ω(Γ_{2}).

We will prove:

Theorem 1.1. Suppose that Γ_{1} and Γ_{2} are two geometrically finite, coinfinite volume discrete groups of isometries of H^{3} with domains of discontinuity Ω(Γ_{1}) and Ω(Γ_{2}) and scattering matrices S_{1}(s) and S_{2}(s), and suppose that there is an orientation-preserving diffeomorphism ψ: Ω(Γ_{1}) → Ω(Γ_{2}) with the following properties:

(i) ψ induces an isomorphism of Γ_{1} and Γ_{2} , and (ii) the operator S_{rel}(s) = S_{1}(s) − Ψ^{∗}S_{2}(s) is a trace-class operator for some s

with Re(s) = 1.

Then ψ is a Möbius transformation, M_{1} and M_{2} are isometric, S_{rel}(s) = 0, and Γ_{1} and Γ_{2} are conjugate in PSL(2, C).

Remarks. 1. There exist quasiconformal maps satisfying (i) but not (ii); we show this below. 2. We will define the pullback of S_{2}(s) in Section 2, where we also specify the Hilbert space in which S_{1}(s) and Ψ^{∗}S_{2}(s) act. 3. The line Re(s) = 1 corresponds to the continuous spectrum of the Laplacian.

Our result uses the following isomorphism theorem of Marden [4] which, as Marden remarks, is an analogue of Mostow’s celebrated rigidity theorem [5], [6].

Theorem 1.2. Suppose that G and H are Kleinian groups such that

(i) G has a finite-sided fundamental polyhedron, (ii) there exists an orientation-preserving homeomorphism f: Ω(G) → Ω(H) which

induces an isomorphism φ: G → H. Then there exists a quasiconformal homeomorphism of the closed ball \overline{B^{3}} → \overline{B^{3}} which induces φ. If f is quasiconformal, f has a quasiconformal extension to ∂B^{3} . If f is conformal, then φ is an inner automorphism.

To prove Theorem 1.1, we use the known singularities of the scattering operator and the trace-class condition to show that the diffeomorphism ψ must be a Möbius transformation on each connected component of Ω_{1} . Marden’s theorem shows that ψ is in fact a single Möbius transformation so that M_{1} and M_{2} are isometric and Ψ^{∗}S_{2}(s) = S_{1}(s). The result should be viewed as a reflection in scattering theory of the rigidity of hyperbolic 3-manifolds.

To appreciate the meaning of Theorem 1.1, let B_{1} and B_{2} be compact Riemann surfaces with fundamental groups G_{1} and G_{2} such that B_{1} and B_{2} are homeomorphic but carry distinct conformal structures. We assume that each has genus 2 or higher. There exists a quasiconformal homeomorphism from B_{1} to B_{2} which lifts to a quasiconformal homeomorphism φ of the Poincaré upper half-plane to itself (see Lehto [3, Theorem V.1.5]). This map induces an isomorphism of G_{1} and G_{2} . By the uniformization theorem, G_{1} and G_{2} can be viewed as discrete subgroups of PSL(2, R) and, by the embedding of PSL(2, R) into PSL(2, C), they can be viewed as discrete groups Γ_{1} and Γ_{2} of isometries of H^{3} . The quotients M_{1} = H^{3}/Γ_{1} and M_{2} = H^{3}/Γ_{2} are cylinders diffeomorphic to R × B_{i}, i = 1, 2. The domains of discontinuity are both a union of two copies of H^{2} and the quasiconformal homeomorphism φ induces a quasiconformal homeomorphism from Ω(Γ_{1}) to Ω(Γ_{2}) which satisfies hypothesis (i) of Theorem 1.1. It can only satisfy hypothesis (ii) if Γ_{1} and Γ_{2} are conjugate in PSL(2, C), i.e., G_{1} and G_{2} are conjugate in PSL(2, R) and B_{1} and B_{2} carry the same conformal structure.

In Section 2, we recall some basic facts about the scattering operator: see for example [7], [8] for further details and references. In Section 3 we give the proof of Theorem 1.1.

Acknowledgements. It is a pleasure to thank Charles Epstein for a helpful conversation. I am also happy to thank Caroline Series for pointing out Marden’s isomorphism theorem and for hospitality at the Mathematical Institute, University of Warwick during part of the time that this work was done.

2. The scattering operator

To define and discuss the scattering operator, it will be convenient to work in the upper half-space model of H^{3} with geometric boundary R^{2} ∪ {∞}. In this setting the domain of discontinuity Ω(Γ) is a subset of R^{2} ∪ {∞}.

Before we define the scattering operator associated to a discrete group Γ, we define the objects on which it acts. A smooth function f on the domain of discontinuity Ω(Γ) is called a Γ-automorphic form of complex weight s if for every γ ∈ Γ the identity

f(γ(x))|γ′(x)|^{s} = f(x)

holds. Here |γ′(x)| denotes the conformal dilation of the Möbius transformation γ at a point x. We denote by Γ(s) the space of smooth, Γ-automorphic forms of complex weight s. If ℱ is any fundamental domain for the action of Γ on Ω(Γ) and f is a form of weight 2, the integral ∫_{ℱ} f(x) dx is independent of the choice of ℱ; thus Γ(2 − s) and Γ(s) are dual spaces under the dual pairing ⟨f, g⟩ = ∫_{ℱ} f(x)g(x) dx. If Re(s) = 1, complex conjugation maps Γ(s) to Γ(2−s) and we may define an inner product on Γ(s) by the formula

(f, g) = ∫_{ℱ} f(x)\overline{g(x)} dx.

Completing Γ(s) in this inner product gives a Hilbert space ℋ (s) on which the scattering operator acts as a unitary operator.

The scattering operator S(s) maps Γ(2−s) to Γ(s) and is defined for Re(s) > 2 by the formula

(S(s)f)(x) = ∫_{ℱ} σ(x, y)f(y) dy,

where σ(x, y) is the distribution kernel

(2.1)  σ(x, y) = ∑_{γ∈Γ} \frac{|γ′(x)|^{s}}{|γ(x) − y|^{2s}}

and dy denotes area measure on R^{2} . The formula

(2.2)  \frac{|γ′(x)||γ′(y)|}{|γ(x) − γ(y)|^{2}} = \frac{1}{|x − y|^{2}} ,

true for Möbius transformations γ, shows that S(s) has the claimed mapping properties. In fact, the sum (2.1) converges in the half-plane Re(s) > δ(Γ) where δ(Γ) is the exponent of convergence for Γ.

It is a deep result of scattering theory that the operator S(s) admits a meromorphic continuation to the complex plane and in particular is well-defined on the line Re(s) = 1 corresponding to the continuous spectrum of Δ. The meromorphically continued operator is a pseudodifferential operator whose Schwarz kernel analytically continues the sum (2.1). It is known to have leading singularity |x − y|^{−2s} plus a smooth remainder (see for example [1], [7]).

To compare two scattering matrices, it is necessary to map Γ_{2}(s) to Γ_{1}(s). In what follows, |Dψ(x)| denotes the Jacobian determinant of the map ψ at x and |γ′(x)| denotes the conformal dilation of the Möbius transformation γ at x. Suppose that f ∈ Γ_{2}(s) and ψ satisfies the hypotheses of Theorem 1. The function g(x) = |Dψ(x)|^{s/2}(f ∘ ψ)(x) belongs to Γ_{1}(s) since, for any γ ∈ Γ_{1} ,

g(γ(x)) = |Dψ(γ(x))|^{s/2} f(ψ(γ(x))) = |Dψ(γ(x))|^{s/2} f(τ(ψ(x)))
= |Dψ(γ(x))|^{s/2} f(ψ(x)) |τ′(ψ(x))|^{−s/2}
= |Dψ(γ(x))|^{s/2} f(ψ(x)) |(ψγψ^{−1})′(ψ(x))|^{−s/2} = |γ′(x)|^{−s}g(x)

where τ = ψγψ^{−1} ∈ Γ_{2} by the hypotheses of Theorem 1. Thus ψ induces a map ψ_{s}^{∗}: Γ_{2}(s) ⟼ Γ_{1}(s) by the formula

ψ_{s}^{∗}f(x) = |Dψ(x)|^{s/2}(f ∘ ψ)(x)

and an inverse map (ψ^{−1})_{s}^{∗}: Γ_{1}(s) ⟼ Γ_{2}(s) by the formula

(ψ^{−1})_{s}^{∗}g(x) = |Dψ^{−1}(x)|^{s/2}(g ∘ ψ^{−1})(x).

Using these maps, we can pull back the scattering operator for Γ_{2} to an operator from Γ_{1}(n − s) to Γ_{1}(s) by the formula

Ψ^{∗}S_{2}(s) = (ψ_{s}^{∗})S_{2}(s)(ψ^{−1})_{2−s}^{∗}.

It is not difficult to see that if S_{2}(s) has integral kernel σ_{2}(x, y), then Ψ^{∗}S_{2}(s) has integral kernel

σ_{2}^{∗}(x, y) = |Dψ(x)|^{s/2}|Dψ(y)|^{s/2}σ_{2}(ψ(x), ψ(y)).

Remark. S(s) can equivalently be viewed as a map between line bundles ℳ_{s}(Γ) over B = Ω(Γ)/Γ defined as quotients of Ω×C by the equivalence relation

(x, z) ∼ (x′, z′) if x′ = γ(x) and z = |γ′(x)|^{s}z′.

Thus S(s) is a pseudodifferential operator on line bundles in the sense of Hörmander [2, Chapter XVIII]. The hypotheses on ψ in Theorem 1.1 say that ψ induces a bundle isomorphism from ℳ_{s}(Γ_{1}) to ℳ_{s}(Γ_{2}).

3. Proof of Theorem 1.1

To prove Theorem 1.1, we prove that: (i) the diffeomorphism ψ must be a conformal diffeomorphism, (ii) this conformal diffeomorphism must be a Möbius transformation in each connected component of Ω_{1} , and (iii) ψ acts as the same Möbius transformation in each connected component.

To carry out step (i), let σ_{1}(x, y) be the Schwarz kernel of S_{1}(s) and let σ_{2}^{∗}(x, y) be the Schwarz kernel of Ψ^{∗}S_{2}(s). Then the leading-order term in the difference σ_{1}(x, y) − σ_{2}^{∗}(x, y) is

\frac{|Dψ(x)|^{s/2}|Dψ(y)|^{s/2}}{|ψ(x) − ψ(y)|^{2s}} − \frac{1}{|x − y|^{2s}}

with the remaining terms being smooth. Since Re(s) = 1, in order for the difference to be nonsingular we must have, for x and y sufficiently close, the equality

\frac{|Dψ(x)|}{|Dψ(x)(x − y)|^{2}} = \frac{1}{|x − y|^{2}} .

This equality implies that, for each x, the matrix |Dψ(x)|^{−1/2}Dψ(x) is an orthogonal matrix: hence ψ is a conformal diffeomorphism. It follows that we may realize ψ as an analytic function of one complex variable. In what follows, we denote this function by ψ(z) and its derivatives in the complex sense by ψ′(z), ψ′′(z), etc.

To carry out step (ii), we Taylor expand the difference

κ(z, w) = \frac{|ψ′(z)|^{s}|ψ′(w)|^{s}}{|ψ(z) − ψ(w)|^{2s}} − \frac{1}{|z − w|^{2s}}

for z and w in a small neighborhood. Since ψ is a diffeomorphism, |ψ′(z)| is bounded below by a strictly positive constant on any compact subset of Ω_{1} . Taylor-expanding ψ(w) and ψ′(w) about w = z, we arrive at the identity

\frac{|ψ′(z)ψ′(w)|}{(ψ(z) − ψ(w))^{2}} = \frac{1}{(z − w)^{2}} (1 + \frac{1}{6}𝒮_{ψ}(z)(z − w)^{2} + O(z − w)^{3}),

where

𝒮_{ψ}(z) = (\frac{ψ′′′(z)}{ψ′(z)} − \frac{3}{2}(\frac{ψ′′(z)}{ψ′(z)})^{2})

is the Schwarzian derivative. It follows from this identity that

κ(w, z) = s(\frac{1}{6}𝒮_{ψ}(z))(z − w)^{2−2s} + O((z − w)^{3−2s}).

Since the function (z −w)^{2−2s} is discontinuous at z = w for any s with Re(s) = 1 and Im(s) ≠ 0, 𝒮_{ψ}(z) = 0, so that ψ acts locally as a Möbius transformation (see for example Lehto [3, Theorem II.1.1]).

A trace-class rigidity theorem for Kleinian groups 257

Now we carry out step (iii). By what we have already shown, ψ acts as a Möbius transformation on each connected component of Ω_{1} . The same holds true in the ball model of H^{3} . Since Γ_{1} is geometrically finite and ψ induces an isomorphism of Γ_{1} and Γ_{2} , it follows from Marden’s theorem that ψ extends to a quasiconformal homeomorphism of the closed unit ball. By continuity, ψ acts by the same Möbius transformation in each connected component, so ψ is the restriction to S^{2} of a Möbius transformation acting on H^{3} . This shows that Γ_{1} and Γ_{2} are conjugate in PSL(2, C) so that M_{1} and M_{2} are isometric.

To see that S_{rel}(s) = 0, we use the formula (2.2) together with the definition of Ψ^{∗}S_{1} .

Remark. The analogous calculation in two dimensions considers two scattering matrices defined for Re(s) = \frac{1}{2} , corresponding to the continuous spectrum of the Laplacian on a surface. Here the diffeomorphism ψ acts between one dimensional manifolds and the trace-class condition yields no constraint on ψ.

References

[1] Froese, R., P. Hislop, and P. Perry: The Laplace operator on hyperbolic threemanifolds with cusps of non-maximal rank. - Invent. Math. 106, 1991, 295–333.

[2] Hörmander, L.: The Analysis of linear partial differential operators, III. Pseudo-differen-tial operators. - Grundlehren Math. Wiss. 274, Springer-Verlag, Berlin–Heidelberg– New York–Tokyo, 1985.

[3] Lehto, O.: Univalent functions and Teichmüller spaces. - Springer-Verlag, New York, 1987.

[4] Marden, A.: The geometry of finitely-generated kleinian groups. - Ann. Math. 99, 1974, 383–462.

[5] Mostow, G.D.: Quasiconformal mappings in n-space and the rigidity of hyperbolic space forms. - Inst. Hautes Études Sci. Publ. Math. 34, 1968, 53–104.

[6] Mostow, G.D.: The rigidity of locally symmetric spaces. - Actes, Congrès intern. Math. 2, 1970, 187–197.

[7] Perry, P.A.: The Laplace operator on a hyperbolic manifold, II. Eisenstein series and the scattering matrix. - J. Reine Angew. Math. 398, 1989, 67–91.

[8] Perry, P.A.: The Selberg Zeta function and a local trace formula for Kleinian groups. - J. Reine Angew. Math. 410, 1990, 116–152.

[9] Perry, P.A.: Relative trace formulas for hyperbolic surfaces of infinite area. - Unpublished.

Received 30 September 1993