Annales Academiæ Scientiarum Fennicæ Mathematica Volumen 21, 1996, 439–462

ON THE MARGULIS CONSTANT FOR KLEINIAN GROUPS, I

F.W. Gehring and G.J. Martin

University of Michigan, Department of Mathematics Ann Arbor, MI 48109, U.S.A.; fgehring@math.lsa.umich.edu University of Auckland, Department of Mathematics Auckland, New Zealand

Dedicated to Seppo Rickman on the occasion of his 60^{th} birthday

Abstract. The Margulis constant for Kleinian groups is the smallest constant c such that for each discrete group G and each point x in the upper half space H^{3} , the group generated by the elements in G which move x less than distance c is elementary. We take a first step towards determining this constant by proving that if ⟨f, g⟩ is nonelementary and discrete with f parabolic or elliptic of order n ≥ 3, then every point x in H^{3} is moved at least distance c by f or g where c = .1829 …. This bound is sharp.

1. Introduction

Let M denote the group of all Möbius transformations of the extended complex plane \overline{C} = C ∪ \{∞\}. We associate with each Möbius transformation

f = \frac{az + b}{cz + d} ∈ M, ad − bc = 1,

the matrix

A = ( \begin{matrix}a & b \\ c & d \end{matrix} ) ∈ SL (2, C)

and set tr (f) = tr (A) where tr (A) denotes the trace of A. Next for each f and g in M we let [f, g] denote the commutator fgf^{−1}g^{−1} . We call the three complex numbers

(1.1) β(f) = tr^{2}(f) − 4,  β(g) = tr^{2}(g) − 4,  γ(f, g) = tr ([f, g]) − 2

1991 Mathematics Subject Classification: Primary 30F40, 20H10, 57N10.

Research supported in part by grants from the U.S. National Science Foundation and the N.Z. Foundation of Research, Science and Technology. The first author wishes to thank the University of Texas at Austin, the University of Auckland and the MSRI at Berkeley (NSF DMS-90222140) for their support.

the parameters of the two generator group ⟨f, g⟩ and write

par (⟨f, g⟩) = (γ(f, g), β(f), β(g)).

These parameters are independent of the choice of representations for f and g and they determine ⟨f, g⟩ up to conjugacy whenever γ(f, g) ≠ 0.

A Möbius transformation f may be regarded as a matrix A in SL (2, C), a conformal self map of \overline{C} or a hyperbolic isometry of H^{3} . There are three different norms, corresponding to these three roles, which measure how much f differs from the identity [4]:

(1.2)  m(f) = ∥A − A^{−1}∥,
(1.3)  d(f) = sup\{q(f(z), z) : z ∈ \overline{C}\},
(1.4)  ρ(f) = h(f(j), j).

Here ∥B∥ denotes the euclidean norm of the matrix B ∈ GL(2, C), q the chordal metric in \overline{C}, j the point (0, 0, 1) ∈ H^{3} and h the hyperbolic metric with curvature −1 in H^{3} . We will refer to m(f), d(f) and ρ(f) as the matrix, chordal and hyperbolic norms of f. All three are invariant with respect to conjugation by chordal isometries.

A subgroup G of M is discrete if

inf{d(f) : f ∈ G, f ≠ id} > 0

or equivalently if

inf{m(f) : f ∈ G, f ≠ id} > 0;

G is nonelementary if it contains two elements with infinite order and no common fixed point and G is Fuchsian if G(H^{2}) = H^{2} where H^{2} is the upper half plane in \overline{C}.

The Margulis constant for Kleinian groups G in M acting on the upper half space H^{3} is the largest constant c = c_{K} with the following property. For each discrete group G and each x ∈ H^{3} , the group generated by

S = {f ∈ G, h(f(x), x) < c}

is elementary. The Margulis constant c_{F} for Fuchsian groups G in M acting on H^{2} is defined exactly as above with H^{3} replaced by H^{2} . That such constants exist follows from [1], [11], [12].

The constant c_{F} was determined by Yamada who showed in [15] that

(1.5)  c_{F} = 2arc sinh (\sqrt{\frac{2 cos(2π/7) − 1}{8 cos(π/7) + 7}} ) = .2629 …

by establishing the following result.

Theorem 1.6. If G = ⟨f, g⟩ is discrete, nonelementary and Fuchsian, then

max{h(f(x), x), h(g(x), x)} ≥ c_{F}

for x ∈ H^{2} . Equality holds only for the case where G is the (2, 3, 7) triangle group and f and g are elliptics of orders 3 and 2.

Culler and Shalen have made important contributions to this problem in the Kleinian case. See [3].

We shall establish in this paper the following partial analog of Theorem 1.6 for the case of Kleinian groups.

Theorem 1.7. If G = ⟨f, g⟩ is discrete and nonelementary and if f is parabolic or elliptic of order n ≥ 3, then

(1.8)  max{h(f(x), x), h(g(x), x)} ≥ c

for x ∈ H^{3} where

(1.9)  c = arc cosh (\frac{2\sqrt{8 + 2\sqrt{5}} − (1 + \sqrt{5} )}{6 − \sqrt{5}} ) = .1829 ….

Inequality (1.9) is sharp and equality holds only if f and g are elliptics of orders 3 and 2.

The following alternative formula

c = 2arc sinh (\sqrt{\frac{4 cos(2π/5) − 1}{4\sqrt{8 cos(2π/5) + 10} + 14}} )

for c is similar to that for the constant c_{F} .

Let

(1.10)  c(3) = arc cosh (\frac{2\sqrt{8 + 2\sqrt{5}} − (1 + \sqrt{5} )}{6 − \sqrt{5}} ) = .1829 …,

(1.11)  c(4) = arc cosh (\frac{\sqrt{6 + 2\sqrt{3}} − \sqrt{3}}{3 − \sqrt{3}} ) = .3453 …,

(1.12)  c(5) = arc cosh (\frac{4(2 + \sqrt{5} − \sqrt{9 − \sqrt{5}} )}{5(\sqrt{5} − 1)} ) = .3401 …,
(1.13)  c(6) = arc cosh (\frac{17}{16}) = .3517 …,
(1.14)  c(n) = arc cosh (\frac{5 − 2 sin^{2}(π/n)}{4 + 2 sin^{2}(π/n)}) ≥ .3343 …

for n ≥ 7 and set

(1.15)  c(∞) = lim_{n→∞} c(n) = arc cosh (5/4) = .6931 ….

Then Theorem 1.7 is a consequence of the following two results.

Theorem 1.16. If G = ⟨f, g⟩ is discrete and nonelementary and if f is elliptic of order n ≥ 3, then

(1.17)  max{h(f(x), x), h(g(x), x)} ≥ c(n)

for x ∈ H^{3} . Inequality (1.17) is sharp for each n ≥ 3 and equality holds only if θ(f) = ±2π/n and f and g are elliptics of orders n ≠ 6 and 2 or of orders 6 and 3.

Theorem 1.18. If G = ⟨f, g⟩ is discrete and nonelementary and if f is parabolic, then

(1.19)  max{h(f(x), x), h(g(x), x)} ≥ c(∞)

for x ∈ H^{3} . Inequality (1.19) is sharp and equality holds only if g is elliptic of order 2.

Given f, g ∈ M \ \{id\}, we let fix (f) denote the set of points in \overline{C} fixed by f. Next if f is nonparabolic, we let ax (f) denote the axis of f, i.e. the closed hyperbolic line in H^{3} with endpoints in fix (f). Finally if f and g are both nonparabolic, we let δ(f, g) denote the hyperbolic distance between ax (f) and ax (g) in H^{3} . Then δ(f, g) > 0 unless ax (f) ∩ ax (g) ≠ ∅.

We prove Theorem 1.16 by considering in §3, §4, §5, respectively, the three cases where f is of order n ≥ 3 and

1. ax (f) ∩ ax (gfg^{−1}) = ∅,

2. ax (f) ∩ ax (gfg^{−1}) ≠ ∅, fix (f) ∩ fix (gfg^{−1}) = ∅,
3. fix (f) ∩ fix (gfg^{−1}) ≠ ∅.

The proof depends on the estimates in [5] for the distance between axes of elliptics and on the diagrams in [8] for the possible values of the commutator parameter for a two generator group with an elliptic generator. Our argument shows also that the extremal groups for which (1.17) holds with equality for some x ∈ H^{3} are unique up to conjugacy.

The proof for Theorem 1.18 is given in §6.

2. Preliminary results

We derive here some formulas and inequalities which will be needed in what follows. First each nonparabolic Möbius transformation f ≠ id in M is conjugate to a transformation of the form ae^{ib} where a > 0 and −π < b ≤ π. Then τ(f) = | log(a)| and θ(f) = b are the translation length and rotation angle of f and it is easy to check that [7]

(2.1)  4 cosh(τ(f)) = |β(f) + 4| + |β(f)|,
(2.2)  4 cos(θ(f)) = |β(f) + 4| − |β(f)|.

The following result gives alternative formulas for the matrix and hyperbolic norms for a nonparabolic Möbius transformation f in terms of the trace parameter β(f) and the axial displacement

δ(f) = h(j, ax (f)),

that is, the hyperbolic distance between j = (0, 0, 1) and the axis of f.

Lemma 2.3. If f ∈ M \ {id} is nonparabolic, then

(2.4)  m(f)^{2} = 2 cosh(2δ(f))|β(f)|,
(2.5)  4 cosh(ρ(f)) = cosh(2δ(f))|β(f)| + |β(f) + 4|.

Proof. Let z_{1}, z_{2} denote the fixed points of f. Then

m(f)^{2} = 2\frac{8 − q(z_{1}, z_{2})^{2}}{q(z_{1}, z_{2})^{2}} |β(f)| = 2 cosh(2δ(f))|β(f)|

by p. 37 and p. 48 in [4], and we obtain

(2.6)  8 cosh(ρ(f)) = m(f)^{2} + 2| tr (f)^{2}| = m(f)^{2} + 2|β(f) + 4|

from p. 46 in [4].

Lemma 2.3 yields a formula for the hyperbolic displacement of a point x ∈ H^{3} under a Möbius transformation f.

Lemma 2.7. If f ∈ M \ {id} is nonparabolic, then

(2.8)  4 cosh(h(x, f(x)) = cosh(2h(x, ax (f)))|β(f)| + |β(f) + 4|

for each x ∈ H^{3} .

Proof. Fix x ∈ H^{3} and let g = ϕfϕ^{−1} where ϕ is a Möbius transformation which maps x onto j. Then β(g) = β(f),

δ(g) = h(j, ax (g)) = h(ϕ(x), ϕ(ax (f))) = h(x, ax (f))

and

ρ(g) = h(g(j), j) = h(ϕ(x), ϕ(f(x))) = h(x, f(x)).

Then (2.8) follows from (2.5) applied to g.

The proof of Theorem 1.16 for the first case in §3 depends on the following two upper bounds for the axial displacement δ(f).

Lemma 2.9. If f ∈ M is elliptic of order n ≥ 3, then

(2.10) cosh(2δ(f)) ≤ \frac{cosh(ρ(f)) − cos^{2}(π/n)}{sin^{2}(π/n)},
(2.11) sinh^{2}(2δ(f)) ≤ \frac{(cosh(ρ(f)) − 1)(cosh(ρ(f)) + 1 − 2 cos^{2}(π/n))}{sin^{4}(π/n)}.

There is equality in (2.10) and (2.11) if and only if θ(f) = ±2π/n.

Proof. Suppose that θ(f) = 2mπ/n where |m| ≤ n/2. Then

β(f) = −4 sin^{2}(mπ/n), β(f) + 4 = 4 cos^{2}(mπ/n)

and thus by (2.5)

cosh(2δ(f)) = \frac{4 cosh(ρ(f)) − |β(f) + 4|}{|β(f)|}

= \frac{cosh(ρ(f)) − cos^{2}(mπ/n)}{sin^{2}(mπ/n)}

≤ \frac{cosh(ρ(f)) − cos^{2}(π/n)}{sin^{2}(π/n)}.

Hence we obtain (2.10), which in turn implies (2.11), with equality in each case if and only if |m| = 1.

Lemma 2.12. If g ∈ M \ {id} is nonparabolic, then

(2.13)  cosh(2δ(g)) ≤ \frac{4 cosh(ρ(g))}{|β(g)|},
(2.14)  sinh(2δ(g)) ≤ \frac{4 sinh(ρ(g))}{|β(g)|}.

There is equality in (2.13) if and only if g is elliptic of order 2 and equality in (2.14) if and only if either g is of order 2 or g is elliptic with δ(g) = 0.

Proof. For (2.13) we see by (2.5) that

cosh(2δ(g)) = \frac{4 cosh(ρ(g)) − |β(g) + 4|}{|β(g)|} ≤ \frac{4 cosh(ρ(f))}{|β(g)|}

with equality if and only if β(g) = −4, that is, if and only if g is of order 2. Next (2.5) implies that

sinh^{2}(2δ(g)) = \frac{N}{|β(g)|^{2}}

where

N = (4 cosh(ρ(g)) − |β(g) + 4|)^{2} − |β(g)|^{2}
= 16 cosh^{2}(ρ(g)) − 8 cosh(ρ(g))|β(g) + 4| + |β(g) + 4|^{2} − |β(g)|^{2}
≤ 16 cosh^{2}(ρ(g)) − 8 cosh(τ(g))|β(g) + 4| + |β(g) + 4|^{2} − |β(g)|^{2}
= 16 cosh^{2}(ρ(g)) − (|β(g) + 4| + |β(g)|)^{2}
≤ 16 sinh^{2}(ρ(g))

by (2.1) and (2.5). This yields (2.14). Equality holds if and only if either g is of order 2 or g is elliptic with δ(g) = 0.

Finally we will use the following two lower bounds for the maximum of the hyperbolic norms ρ(f) and ρ(g) in the proof of Theorem 1.16 for the second and third cases in §4 and §5.

Lemma 2.15. If f, g ∈ M \ {id} and if ρ = max{ρ(f), ρ(g)}, then
(2.16)  8 cosh(ρ) ≥ M

where

M = |β(f) + 4| + |β(g) + 4| + \sqrt{m(f)^{2}m(g)^{2} + (|β(f) + 4| − |β(g) + 4|)^{2}}.

In addition,

(2.17)  m(f)^{2}m(g)^{2} ≥ 2(|4γ(f, g) + β(f)β(g)| + |4γ(f, g)| + |β(f)β(g)|).

There is equality in (2.16) if and only if ρ(f) = ρ(g) and in (2.17) for nonparabolic f and g if and only if δ(f) = δ(g) = δ(f, g)/2.

Proof. Let t = cosh(ρ). Then

8 cosh(ρ(f)) − 2|β(f) + 4| = m(f)^{2}, 8 cosh(ρ(g)) − 2|β(g) + 4| = m(g)^{2}

by (2.6). Hence

(2.18)  (8t − 2|β(f) + 4|)(8t − 2|β(g) + 4|) ≥ m(f)^{2}m(g)^{2}

and we obtain

8t ≥ |β(f) + 4| + |β(g) + 4| + \sqrt{m(f)^{2}m(g)^{2} + (|β(f) + 4| − |β(g) + 4|)^{2}}

with equality whenever ρ(f) = ρ(g).

Next if f or g is parabolic, then β(f)β(g) = 0 and

\begin{aligned}m(f)^{2}m(g)^{2} & ≥ 16|γ(f, g)| \\ & = 2(|4γ(f, g) + β(f)β(g)| + |4γ(f, g)| + |β(f)β(g)|) \end{aligned}

by Theorem 2.7 in [4]. Otherwise choose x ∈ ax (f) and y ∈ ax (g) so that δ(f) = h(x, j) and δ(g) = h(x, j). Then

δ(f, g) ≤ h(x, y) ≤ h(x, j) + h(y, j) = δ(f) + δ(g)

and hence by (2.4) and Lemma 4.4 of [5],

m(f)^{2}m(g)^{2} = 4 cosh(2δ(f)) cosh(2δ(g))|β(f)β(g)|
≥ 4 cosh^{2}(δ(f) + δ(g))|β(f)β(g)|
≥ 4 cosh^{2}(δ(f, g))|β(f)β(g)|
= 2(cosh(2δ(f, g)) + 1)|β(f)β(g)|
= 2(|4γ(f, g) + β(f)β(g)| + |4γ(f, g)| + |β(f)β(g)|)

with equality throughout if and only if δ(f) = δ(g) = δ(f, g)/2.

Lemma 2.19. If f, g ∈ M \ {id} and if ρ = max{ρ(f), ρ(g)}, then

(2.20)  |β(g) + 4| ≤ 4 cosh(ρ) − \frac{4|γ(f, g)|}{4 cosh(ρ) − |β(f) + 4|}.

Proof. Theorem 2.7 of [4] implies that

m(f)^{2}m(g)^{2} ≥ 16|γ(f, g)|.

Hence

(4 cosh(ρ) − |β(f) + 4|)(4 cosh(ρ) − |β(g) + 4|) ≥ 4|γ(f, g)|

by (2.18) and we obtain (2.20).

3. Case where ax (f) ∩ ax (gfg^{−1}) = ∅

We shall establish here in Theorem 3.2 a sharp version of Theorem 1.16 for the case where f is of order n ≥ 3 with

ax (f) ∩ ax (gfg^{−1}) = ∅.

In this case,

δ = δ(f, gfg^{−1}) > 0.

Then the fact f and gfg^{−1} are elliptic of order n ≥ 3 allows us to combine Lemmas 2.9 and 2.12 with the sharp lower bound b(n) for δ in [5] to obtain a lower bound for the maximal hyperbolic displacement of each point x in H^{3} under f and g.

For convenience of notation, for n ≥ 3 we set

(3.1)  d(n) = \{ \begin{matrix}c(n) & if n ≠ 6, \\ arc cosh (6 − \sqrt{24} ) = .4457 … & if n = 6 \end{matrix}

since the lower bound in Theorem 3.2 is greater than that in Theorem 1.16 when n = 6.

Theorem 3.2. If ⟨f, g⟩ is discrete, if f is elliptic of order n ≥ 3 and if δ(f, gfg^{−1}) > 0, then

(3.3)  max{h(f(x), x), h(g(x), x)} ≥ d(n)

for x ∈ H^{3} . Inequality (3.3) is sharp for each n ≥ 3 and equality holds only if θ(f) = ±2π/n and g is elliptic of order 2.

Proof. Fix x ∈ H^{3} and let f_{1} = ϕfϕ^{−1} and g_{1} = ϕgϕ^{−1} where ϕ is a Möbius transformation which maps x onto j. Then ⟨f_{1}, g_{1}⟩ satisfies the hypotheses of Theorem 3.2,

max{h(f(x), x), h(g(x), x)} = max{h(f_{1}(j), j), h(g_{1}(j), j)}

and hence it suffices to establish (3.3) for the case where x is the point j. Next choose x ∈ ax (f) and y ∈ ax (g) so that

δ(f) = h(x, j), δ(g) = h(y, j).

Then g(x) ∈ ax (gfg^{−1}),

δ = δ(f, gfg^{−1}) ≤ h(x, g(x)),

and thus

4 cosh(δ) ≤ 4 cosh(h(x, g(x))) = cosh(2h(x, ax (g)))|β(g)| + |β(g) + 4|

by Lemma 2.7. Next

h(x, ax (g)) ≤ h(x, y) ≤ h(x, j) + h(y, j) = δ(f) + δ(g)

by the triangle inequality and we obtain

(3.4)  4 cosh(δ) ≤ cosh(2δ(f) + 2δ(g))|β(g)| + |β(g) + 4| = R_{1} + R_{2}

448 F.W. Gehring and G.J. Martin where

R_{1} = cosh(2δ(f)) cosh(2δ(g))|β(g)| + |β(g) + 4|

(3.5) ≤ (cosh(2δ(g))|β(g)| + |β(g) + 4|) cosh(2δ(f))

= 4 cosh(ρ(g)) cosh(2δ(f)),

(3.6)  R_{2} = sinh(2δ(g))|β(g)| sinh(2δ(f))

by (2.5) of Lemma 2.3.

Since ax (f) ∩ ax (gfg^{−1}) = ∅,

ρ = max{ρ(f), ρ(g)} > 0, t = cosh(ρ) > 1.

Then by Lemma 2.9,

(3.7)  cosh(2δ(f)) sin^{2}(π/n) ≤ t − cos^{2}(π/n),
(3.8)  sinh(2δ(f)) sin^{2}(π/n) ≤ \sqrt{(t − 1)(t + 1 − 2 cos^{2}(π/n))}

with equality in (3.7) and (3.8) only if θ(f) = ±2π/n. Next by Lemma 2.12,

(3.9)  sinh(2δ(g))|β(g)| ≤ 4\sqrt{t^{2} − 1}

with equality only if g is of order 2. Then (3.4) through (3.9) imply that

(3.10)  cosh(δ) sin^{2}(π/n) ≤ ϕ(t, n),

where

(3.11)  ϕ(t, n) = t(t − cos^{2}(π/n)) + (t − 1)\sqrt{(t + 1)(t + 1 − 2 cos^{2}(π/n))},

and that (3.10) holds with equality only if θ(f) = ±2π/n and g is of order 2.

For n ≥ 3 let

ψ(n) = cosh(b(n)) sin^{2}(π/n)

where b(n) denotes the minimum distance between disjoint axes of elliptics of order n in a discrete group. Then

(3.12)  ψ(n) = \{ \begin{matrix}(1 + \sqrt{5} )/4 = .8090 … & if n = 3, \\ (1 + \sqrt{3} )/4 = .6803 … & if n = 4, \\ .5 & if n = 5, \\ .5 & if n = 6, \\ cos^{2}(π/n) − .5 ≥ .3117 … & if n ≥ 7 \end{matrix}

by Theorem 4.18 in [5]. Next ϕ(t, n) is increasing in t for 1 ≤ t < ∞,

(3.13)  ϕ(1, n) = sin(π/n)^{2} < ψ(n) ≤ cosh(δ) sin^{2}(π/n) ≤ ϕ(t, n)

by (3.10) and we obtain

(3.14)  ρ = arc cosh (t) ≥ arc cosh (t(n))

where t(n) is the unique root of the equation ϕ(t, n) = ψ(n). An elementary but technical calculation shows that

arc cosh (t(n)) = d(n)

and hence that

(3.15)  max{ρ(f), ρ(g)} = ρ ≥ d(n)

with equality only if θ(f) = ±2π/n and g is of order 2. This completes the proof for inequality (3.3).

To show that (3.3) is sharp, fix n ≥ 3. Then by §8 in [5] we can choose elliptics f and g of orders n and 2 such that ⟨f, g⟩ is discrete with

2δ(f, g) = δ(f, gfg^{−1}) = b(n), θ(f) = 2π/n.

Choose x ∈ ax (f) and y ∈ ax (g) so that h(x, y) = δ(f, g). By means of a preliminary conjugation we may assume that x and y lie on the j -axis in H^{3} with the point j situated so that

cosh(2δ(f)) sin^{2}(π/n) + cos^{2}(π/n) = cosh(2δ(g)).

Then β(f) = −4 sin^{2}(π/n), β(g) = −4 and

4 cosh(ρ(f)) = cosh(2δ(f))|β(f)| + |β(f) + 4|
= 4(cosh(2δ(f)) sin^{2}(π/n) + cos^{2}(π/n))
= 4 cosh(2δ(g))
= cosh(2δ(g))|β(f)| + |β(g) + 4|
= 4 cosh(ρ(g)).

Hence if we set

t = cosh(ρ(f)) = cosh(ρ(g)),

then

cosh(2δ(f)) = \frac{t − cos^{2}(π/n)}{sin^{2}(π/n)}, cosh(2δ(g)) = t

and we obtain

ψ(n) = cosh(b(n)) sin^{2}(π/n)
= cosh(2δ(f, g)) sin^{2}(π/n)
= cosh(2δ(f) + 2δ(g)) sin^{2}(π/n)
= t(t − cos^{2}(π/n)) + (t − 1)\sqrt{(t + 1)(t + 1 − 2 cos^{2}(π/n))}
= ϕ(t, n).

Thus

max{ρ(f), ρ(g)} = arc cosh (t) = arc cosh (t(n)) = d(n)

and (3.3) holds with equality.

4. Case where ax (f) ∩ ax (gfg^{−1}) ≠ ∅

We next prove Theorem 1.16 for the case where f is of order n ≥ 3 with

ax (f) ∩ ax (gfg^{−1}) ≠ ∅, fix (f) ∩ fix (gfg^{−1}) = ∅.

Then ⟨f, gfg^{−1}⟩ is an elliptic group, that is, either the cyclic group C_{n}, the dihedral group D_{n}, the tetrahedral group A_{4} , the octahedral group S_{4} or the icosahedral group A_{5} . The hypothesis that f and gfg^{−1} have no common fixed point implies that ⟨f, gfg^{−1}⟩ ≠ C_{n} while the fact that f and gfg^{−1} are both of order n ≥ 3 implies that ⟨f, gfg^{−1}⟩ ≠ D_{n}. The remaining three cases are considered in the following result.

Theorem 4.1. If ⟨f, g⟩ is discrete, if f is elliptic of order n ≥ 3 and if ⟨f, gfg^{−1}⟩ is one of the three groups A_{4} , S_{4} , A_{5} , then either ⟨f, g⟩ is itself one these three groups or

(4.2)  max{h(f(x), x), h(g(x), x)} > c(n)

for x ∈ H^{3} .

We will make use of the following results concerning the parameters of Möbius transformations in the proof for Theorem 4.1.

Lemma 4.3. Suppose that f, g ∈ M \ {id}. Then

(4.4)  γ(f, g^{2}) = γ(f, g)(β(g) + 4),  β(g^{2}) = β(g)(β(g) + 4)

and

(4.5)  γ(f, gfg^{−1}) = γ(f, g)(γ(f, g) − β(f)).

If f is of order 2, then

(4.6)  β(fg) = γ(f, g) − β(g) − 4.

If f and g have disjoint fixed points, then there exists f̃ ∈ M of order 2 such that ⟨f̃, g⟩ is discrete whenever ⟨f, g⟩ is and such that

(4.7)  γ(f̃, g) = β(g) − γ(f, g).

Proof. The identities in (4.4) and (4.5) follow from direct calculation and from Lemma 2.1 in [5]. Next if f is of order 2, then

γ(f, g) = tr ([f, g]) − 2 = tr (g)^{2} + tr (fg)^{2} − 4 = β(g) + β(fg) + 4

by the Fricke identity and we obtain (4.6). Finally for (4.7) set f̃ = ϕf where ϕ is the Lie product of f and g which conjugates f and g to their inverses [9] and [10]. See also Lemma 2.29 of [5].

We shall also need the following list of possible parameters for the groups A_{4}, S_{4}, A_{5} with conjugate elliptic generators.

Lemma 4.8. Suppose that f and h are conjugate elliptics of order n ≥ 3. If ⟨f, h⟩ = A_{4} , then

(4.9)  par (⟨f, h⟩) = (−2, −3, −3).

If ⟨f, h⟩ = S_{4} , then

(4.10)  par (⟨f, h⟩) = (−1, −2, −2).

If ⟨f, h⟩ = A_{5} , then

(4.11)  par (⟨f, h⟩) = (−1, −3, −3)

or

(4.12)  par (⟨f, h⟩) = (−.381 …, −1.381 …, −1.381 …)

or

(4.13)  par (⟨f, h⟩) = (−2.618 …, −3.618 …, −3.618 …).

Proof for Theorem 4.1. Suppose that ⟨f, g⟩ satisfies the hypotheses of Theorem 4.1 and that ⟨f, g⟩ is not any of the groups A_{4} , S_{4} , A_{5} . We must prove that (4.2) holds for each x ∈ H^{3} . As in the proof of Theorem 3.2, it suffices to do this for the case where x is the point j. Let

ρ = max{ρ(f), ρ(g)},  t = cosh(ρ),  β = β(g).

Then

(4.14)  |β + 4| + |β| = |β(g) + 4| + |β(g)| ≤ 4 cosh(ρ(g)) ≤ 4t

by (2.5). We will show that ρ > c(n) by considering separately the three cases where ⟨f, gfg^{−1}⟩ is A_{4} , S_{4} or A_{5} .

Case where ⟨f, gfg^{−1}⟩ = A_{4}

By (4.9),

par (⟨f, gfg^{−1}⟩) = (−2, −3, −3), par (⟨f, g⟩) = (γ, −3, β)

where γ(γ + 3) = −2 by (4.5). Hence γ = −1 or γ = −2.

Suppose that γ = −1. Then β(f) = −3 and

(4.15)  |β + 4| ≤ 4t − \frac{4}{4t − 1}

by Lemma 2.19. Next by Lemma 4.3, ⟨f, g^{2}⟩ is discrete with commutator parameter

γ̃ = γ(f, g^{2}) = γ(f, g)(β(g) + 4) = −β − 4.

Since f is of order 3, it follows from §5.13 of [5] that

γ̃ ∈ {−3, −2.618 …, −2, −1, −.381 …, 0}

or that

|γ̃ + 3| + |γ̃| ≥ \sqrt{5} + 1.

In the first case

β ∈ {−1, −1.381 …, −2, −3, −3.618 …, −4}

and ⟨f, g⟩ is S_{4} or A_{5} unless β = −1 in which case Lemma 2.15 implies that ρ ≥ .3418 > c(3). In the second case we have

(4.16)  |β + 1| + |β + 4| ≥ \sqrt{5} + 1.

Then (4.14), (4.15) and (4.16) imply that ρ ≥ .203 > c(3). See Figure 1.

Suppose next that γ = −2. Then

(4.17)  |β + 4| ≤ 4t − \frac{8}{4t − 1}

by Lemma 2.19. By Lemma 4.3, there exists f̃ of order 2 such that ⟨f̃, g⟩ is discrete with

γ(f̃g, g) = γ(f̃, g) = β(g) − γ(f, g) = β + 2

and

β(f̃g) = γ(f̃, g) − β(g) − 4 = −2.

Hence γ̃ = β + 2 is the commutator parameter of the discrete group ⟨f̃g, g⟩ with a generator of order 4 and by §5.9 of [5] either γ̃ ∈ {−2, −1, 0} or

|γ̃ + 2| + |γ̃| ≥ \sqrt{3} + 1.

In the first case, either β ∈ {−4, −3} and ⟨f, g⟩ is A_{4} or β = −2 and ρ ≥ .428 > c(3) by Lemma 2.15. In the second case,

(4.18)  |β + 4| + |β + 2| ≥ \sqrt{3} + 1

and we obtain ρ ≥ .389 > c(3) from (4.14), (4.17) and (4.18). See Figure 2.

Case where ⟨f, gfg^{−1}⟩ = S_{4}

By (4.10)

par (⟨f, gfg^{−1}⟩) = (−1, −2, −2), par (⟨f, g⟩) = (γ, −2, β)

where γ(γ + 2) = −1 by (4.5). Hence γ = −1 and we obtain

(4.19)  |β + 4| ≤ 4t − \frac{2}{2t − 1}

from Lemma 2.19. Next ⟨f, g^{2}⟩ is discrete with commutator parameter γ̃ = −β−4 and f is of order 4. Hence γ̃ ∈ {−2, −1, 0} or

|γ̃ + 2| + |γ̃| ≥ \sqrt{3} + 1.

by §5.9 of [5]. In the first case, β ∈ {−2, −3, −4} and ⟨f, g⟩ is S_{4} . Otherwise

(4.20)  |β + 2| + |β + 4| ≥ \sqrt{3} + 1.

Next by [8]

γ̃ ∈ {−2.618 …, −2.419 … ± .606i …, −1.877 … ± .744i …}

or |γ̃ + 2| > .8. In the first case

β̃ ∈ {−1.381 …, −1.580 … ± .606i …, −2.122 … ± .744i …}

and either ⟨f, g⟩ is not discrete or ρ ≥ .405 > c(4) by Lemma 2.15. Otherwise

(4.21)  |β + 2| > .8.

Finally if we combine (4.14), (4.19), (4.20) and (4.21), we obtain ρ ≥ .352 > c(4).

Case where ⟨f, gfg^{−1}⟩ = A_{5}

In this case we have the following three possibilities given in (4.11), (4.12) and (4.13). If (4.11) holds, then

par (⟨f, gfg^{−1}⟩) = (−1, −3, −3), par (⟨f, g⟩) = (γ, −3, β)

where γ(γ + 3) = −1 by (4.5); hence γ = −.381 … or γ = −2.618 . . ..

If γ = −.381 …, then ⟨f, g^{2}⟩ is discrete with commutator parameter γ̃ = γ(β + 4) and a generator of order 3. Hence by Theorem 3.4 and Lemmas 2.29 and 6.1 of [5], either γ̃ ∈ {−1, −.381 …, 0} whence β ∈ {−1.381 …, −3, −4} or

|γ̃ + 1| ≥ .618 …,  |γ̃ + .381 … | ≥ .381 …,  |γ̃| ≥ .246 …

whence

(4.22)  |β + 1.381 … | ≥ 1.618,  |β + 3| ≥ 1,  |β + 4| ≥ .646.

Since ⟨f, g⟩ is A_{5} if β ∈ {−1.381 …, −4} and not discrete if β = −3, we obtain (4.22). This and (4.14) imply that ρ ≥ .481 > c(3). See Figure 3.

Figure 3

If γ = −2.618 …, then
(4.23)  |β + 4| ≤ 4t − \frac{10.472}{4t − 1}

by Lemma 2.19. By Lemma 4.3 there exists f̃ of order 2 such that ⟨f̃g, g⟩ is discrete with commutator parameter γ̃ = β + 2.618 … and β(f̃g) = −1.381 …. Since f̃g is of order 5, γ̃ ∈ {−1.381 …, −1, −.381 …, 0} or

|γ̃ + 1.381 … | + |γ̃| ≥ 2

by §5.4 of [5]. In the first case, β ∈ {−4, −3.618 …, −3, −2.618 …} and either ⟨f, g⟩ is A_{5} or not discrete or ρ ≥ .341 > c(3) by Lemma 2.15. In the second case

(4.24)  |β + 4| + |β + 2.618 … | ≥ 2

and we obtain ρ ≥ .395 > c(3) from (4.14), (4.23) and (4.24).

Next if (4.12) holds, then

par (⟨f, gfg^{−1}⟩) = (−.381 …, −1.381 …, −1.381 …)

and

par (⟨f, g⟩) = (γ, −1.381 …, β)

where γ = −.381 … or γ = −1.

If γ = −.381 …, then

(4.25)  |β + 4| ≤ 4t − \frac{1.527}{4t − 2.618}

by Lemma 2.19 and ⟨f, g^{2}⟩ is discrete with commutator parameter γ̃ = γ(β + 4) and f of order 5. Then as above, γ̃ ∈ {−1.381 …, −1, −.381 …, 0} or

|γ̃ + 1.381 … | + |γ̃| ≥ 2.

In the first case β ∈ {−.381 …, −1.381 …, −3, −4} and ⟨f, g⟩ is A_{5} or not discrete. In the second case

(4.26)  |β + .381 … | + |β + 4| ≥ \sqrt{5} + 3

and (4.14), (4.25) and (4.26) imply that ρ ≥ .689 > c(5).

When γ = −1,

(4.27)  |β + 4| ≤ 4t − \frac{4}{4t − 2.618}

and γ̃ = −β − 4 is the commutator parameter of ⟨f, g^{2}⟩ where f is of order 5. Again γ̃ ∈ {−1.381 …, −1, −.381 …, 0} or

|γ̃ + 1.381 … | + |γ̃| ≥ 2.

Hence β ∈ {−2.618 …, −3, −3.618 …, −4} or

(4.28)  |β + 2.618 … | + |β + 4| ≥ 2.

In the first case ⟨f, g⟩ is A_{5} or not discrete; hence we obtain (4.28). Next from [8] it follows that either γ̃ ∈ {−2, −1.5 ± .606i …} or that

|γ̃ + 1.381 … | ≥ .8.

In the first case β ∈ {−2, −2.5 ± .606i …} and ρ ≥ .348 > c(5). In the second case

(4.29)  |β + 2.618 … | ≥ .8

and we conclude from (4.14), (4.27), (4.28) and (4.29) that ρ > .364 > c(5). See Figure 4.

Finally (4.13) implies that

par (⟨f, g⟩) = (γ, −3.618 …, β)

where γ = −1 or γ = −2.618 ….

If γ = −1 …, then

par (⟨f^{2}, g⟩) = (−.381 …, −1.381 …, β),

⟨f^{2}, g⟩ is nonelementary, and we conclude that 2ρ ≥ .689 > 2c(5) from what was proved above.

If γ = −2.618 …, then

(4.30)  |β + 4| ≤ 4t − \frac{10.472}{4t − .381}

by Lemma 2.19. By Lemma 4.3 we can choose f̃ so that ⟨f̃g, g⟩ is discrete with commutator parameter γ̃ = β + 2.618 … and f̃g of order 5. Then as above, [5] and [8] imply that

β ∈ {−4, −3.618 …, −3, −2.618 …, −2.5 ± .606i … − 2},

or that

(4.31)  |β + 2.618 … | + |β + 4| ≥ 2,  |β + 2.618 … | ≥ .8.

In the first case, ⟨f, g⟩ is A_{5} or not discrete or ρ ≥ .445 > c(5). In the second case case, (4.14), (4.30) and (4.31) imply that ρ ≥ .383 > c(5).

5. Case where fix (f) ∩ fix (gfg^{−1}) ≠ ∅

We establish here Theorem 1.16 for the case where f is an elliptic of order n ≥ 3 and

ax (f) ∩ ax (gfg^{−1}) ≠ ∅, fix (f) ∩ fix (gfg^{−1}) ≠ ∅.

Then since ⟨f, g⟩ is nonelementary,

fix (f) ∩ fix (g) = ∅  and  γ(f, g) ≠ 0.

Hence [f, g] is parabolic and n = 3, 4, 6. See [2] or [13]. Next

0 = γ(f, gfg^{−1}) = γ(f, g)(γ(f, g) − β(f))

by (4.5) of Lemma 4.3 and thus

(5.1)  par (⟨f, g⟩) = (β(f), β(f), β(g)).

Theorem 1.16 follows for the case considered here from the following result.

Theorem 5.2. If ⟨f, g⟩ is discrete and not dihedral, if f is elliptic of order n ≥ 3 and if

par (⟨f, g⟩) = (β(f), β(f), β(g)),

then

(5.3)  max{h(f(x), x), h(g(x), x)} ≥ c(n)

for x ∈ H^{3} . Equality (5.3) is sharp only when n = 6 and g is of order 3.

Proof. Again it suffices to establish (5.3) for the case where x = j. Let

ρ = max{ρ(f), ρ(g)},  t = cosh(ρ),  β = β(g).

From the above discussion we see that n is either 3, 4 or 6. We establish (5.3) by considering each of these cases separately.

Case n = 3

In this case

par (⟨f, g⟩) = (−3, −3, β)

and

(5.4)  |β + 4| ≤ 4t − \frac{12}{4t − 1}

by Lemma 2.19. Next by Lemma 4.3 there exists f̃ such that ⟨f̃, g⟩ is discrete with

γ̃ = γ(f̃, g) = β(g) − γ(f, g) = β + 3

and

β(f̃g) = γ(f̃, g) − β(g) − 4 = −1.

Thus ⟨f̃g, g⟩ has a generator of order 6 and either γ̃ ∈ {−1, 0} or

|γ̃ + 1| + |γ̃| ≥ 2

by §5.3 of [5]. In the first case, β = −4 and ⟨f, g⟩ is the dihedral group D_{3} or β = −3 and ρ ≥ .477 > c(3) by Lemma 2.15. Otherwise

(5.5)  |β + 4| + |β + 3| ≥ 2

and this together with (5.4) and (4.14) implies that ρ ≥ .5 > c(3).

Case n = 4

Here

par (⟨f, g⟩) = (−2, −2, β)

and

(5.6)  |β + 4| ≤ 4t − \frac{8}{4t − 2}

by Lemma 2.19. By Lemma 4.3 there exists f̃ such that ⟨f̃, g⟩ is discrete with

γ̃ = γ(f̃, g) = β + 2, β(f̃g) = −2.

Then γ̃ is the commutator parameter of a two generator group with a generator of order 4 and hence γ̃ ∈ {−2, −1, 0} or

|γ̃ + 2| + |γ̃| ≥ \sqrt{3} + 1

by §5.9 of [5]. In the first case β = −4 and ⟨f, g⟩ is D_{4} or β ∈ {−3, −2} and ρ ≥ .428 > c(4) by Lemma 2.15. Otherwise

(5.7)  |β + 4| + |β + 2| ≥ \sqrt{3} + 1

which together with (5.6) and (4.14) implies that ρ ≥ .502 > c(4).

Case n = 6

Finally in this case

par (⟨f, g⟩) = (−1, −1, β)

while

(5.8)  |β + 4| ≤ 4t − \frac{4}{4t − 3}

by Lemma 2.19. Next ⟨f, g^{2}⟩ is discrete with a generator of order 6 and

γ̃ = γ(f, g^{2}) = γ(f, g)(β(f) + 4) = −β − 4

by Lemma 4.3. Thus γ̃ = −1, γ̃ = 0 or

|γ̃ + 1| + |γ̃| ≥ 2

by §5.3 of [5]. In the second case β = −4 and ⟨f, g⟩ is D_{6} . In the third case

(5.9)  |β + 4| + |β + 3| ≥ 2

and this with (5.8) and (4.14) implies that ρ ≥ .394.

It remains to consider the first case where β = −3 and where ⟨f, g⟩ is discrete and nonelementary by [14]. Then

(5.10)  4 cosh(ρ) = max{cosh(2δ(f)) + 3, 3 cosh(2δ(g)) + 1}

by (2.5). Next if we choose x ∈ ax (f) and y ∈ ax (g) so that δ(f) = h(x, j) and δ(g) = h(y, j), then

δ(f, g) ≤ h(x, y) ≤ h(x, j) + h(y, j) = δ(f) + δ(g)

and hence

(5.11)  5/3 = cosh(2δ(f, g)) ≤ cosh(2δ(f) + 2δ(g))

by Lemma 4.4 in [5]. It is then easy to verify from (5.10) and (5.11) that

ρ ≥ arc cosh (17/16) = c(6)

with equality if x and y lie in the j -axis and are situated so that

cosh(2δ(f)) + 3 = 3 cosh(2δ(g)) + 1.

6. Case where f is parabolic

Finally we establish Theorem 1.18, and hence complete the proof of Theorem 1.7, by showing that if ⟨f, g⟩ is discrete and nonelementary and if f is parabolic, then

(6.1)  max{h(f(x), x), h(g(x), x)} ≥ arc cosh (5/4) = c(∞)

for each x ∈ H^{3} . As before it suffices to establish (6.1) for the case where x = j.

Let

ρ = max{ρ(f), ρ(g)}, t = cosh(ρ).

Since j is fixed by chordal isometries, we may assume that f(∞) = ∞ and hence that f and g can be represented by the matrices

A = ( \begin{matrix}1 & u \\ 0 & 1 \end{matrix} )  and  B = ( \begin{matrix}a & b \\ c & d \end{matrix} )

where ad = 1 + bc. Then by Theorem 4.21 of [2],

(6.2)  |u|^{2} + 2 = 2 cosh(ρ(f)) ≤ 2t

and

(6.3)  \begin{aligned}2 + (|c| − |b|)^{2} & ≤ 2|1 + bc| + |c|^{2} + |b|^{2} \\ & ≤ |a|^{2} + |d|^{2} + |b|^{2} + |c|^{2} \\ & = 2 cosh(ρ(g)) ≤ 2t. \end{aligned}

In addition,

(6.4)  |c|^{2}|u|^{2} = |γ(f, g)|^{2} ≥ 1

by the Shimizu–Leutbecher inequality; see II.C in [13].

Suppose that t < 5/4. Then (6.2), (6.4), (6.3) imply, respectively, that

|u| < 1/\sqrt{2}.  |c| > \sqrt{2},  |b| > 1/\sqrt{2}

and hence that

5/2 < |c|^{2} + |b|^{2} ≤ 2 cosh(ρ(g)) ≤ 2t < 5/2,

a contradiction. Thus t ≥ 5/4 and we obtain inequality (6.1).

Suppose next that t = 5/4. Then

|u| ≤ 1/\sqrt{2}.  |c| ≥ \sqrt{2},  |b| ≥ 1/\sqrt{2}

as above and

5/2 ≤ |c|^{2} + |b|^{2} ≤ |a|^{2} + |d|^{2} + |b|^{2} + |c|^{2} = 2 cosh(ρ(g)) ≤ 5/2.

Hence in this case, a = d = 0 and we conclude that (6.1) holds with equality only if g is of order 2.

Finally if f and g are as above with a = d = 0, u = b = 1/\sqrt{2} and c = −\sqrt{2}, then ⟨f, g⟩ is conjugate to the modular group, and hence discrete and nonelementary, with

cosh(ρ(f)) = cosh(ρ(g)) = 5/4.

Thus inequality (6.1) is sharp.

References

[1] Apanasov, B.N.: A universal property of Kleinian groups in the hyperbolic metric. - Dokl. Akad. Nauk SSSR 225, 1975, 1418–1421.

[2] Beardon, A.F.: The Geometry of Discrete Groups. - Springer-Verlag, 1983.

[3] Culler, M., and P. Shalen: Paradoxical decompositions, 2-generator Kleinian groups, and volumes of hyperbolic 3-manifolds. - J. Amer. Math. Soc. 5, 1992, 231–288.

462 F.W. Gehring and G.J. Martin

[4] Gehring, F.W., and G.J. Martin: Inequalities for Möbius transformations and discrete groups. - J. Reine Angew. Math. 418, 1991, 31–76.

[5] Gehring, F.W., and G.J. Martin: Commutators, collars and the geometry of Möbius groups. - J. Analyse Math. 63, 1994, 175–219.

[6] Gehring, F.W., and G.J. Martin: On the minimal volume hyperbolic 3-orbifold. - Math. Res. Letters 1, 1994, 107–114.

[7] Gehring, F.W., and G.J. Martin: Chebyshev polynomials and discrete groups. - Proceedings of the International Conference on Complex Analysis at the Nankai Institute of Mathematics 1992, International Press, 1994, 114–125.

[8] Gehring, F.W., and G.J. Martin: Commutator spectra for discrete groups with an elliptic generator. - In preparation.

[9] Jørgensen, T.: Compact 3-manifolds of constant negative curvature fibering over the circle. - Ann. Math. 106, 1977, 61–72.

[10] Jørgensen, T.: Comments on a discreteness condition for subgroups of SL (2, C). - Canad. J. Math. 31, 1979, 87–92.

[11] Kaz̆dan, D.A., and G.A. Margulis: A proof of Selberg’s conjecture. - Math. USSR-Sb. 4, 1968, 147–152.

[12] Marden, A.: Universal properties of Fuchsian groups in the Poincaré metric. - Ann. Math. Stud. 79, 1974, 315–339.

[13] Maskit, B.: Kleinian Groups. - Springer-Verlag, 1988.

[14] Maskit, B.: Some special 2-generator Kleinian groups. - Proc. Amer. Math. Soc. 106, 1989, 175–186.

[15] Yamada, A.: On Marden’s universal constant of Fuchsian groups. - Kodai Math. J. 4, 1981, 266–277.

[16] Yamada, A.: On Marden’s universal constant of Fuchsian groups, II. - J. Analyse Math. 41, 1982, 234–248.

Received 4 July 1995