Lobachevskii Journal of Mathematics Vol. 3, 1999, 221–241 \begin{matrix}c \\ ◯ \end{matrix}V. F. Molchanov

V. F. Molchanov

REPRESENTATIONS OF PSEUDO-UNITARY GROUPS ASSOCIATED WITH A CONE

(submitted by B.N.Shapukov)

Abstract. We study representations of the pseudo-unitary group SU(p, q), p, q ≥ 2, associated with an isotropic cone.

At present harmonic analysis on semisimple symmetric spaces (pseudo-Rie-mannian, generally speaking) is studied intensively. An important subclass of such spaces is formed by hyperbolic spaces (real, complex, quaternionic) [14], [9], [3]. To construct harmonic analysis on hyperbolic spaces, one needs the so-called representations associated with the isotropic cone. In this paper we study such representations for complex hyperbolic spaces G/H, where G = SU(p, q), H = U(p, q − 1), p ≥ 2, q ≥ 2. The subgroup H has an onedimensional centre, so that these spaces G/H are semi-Kählerian symmetric spaces (in terminology of Berger [1]). This means that G/H has a G-invariant complex structure and a G-invariant pseudo-Hermitian metric. We follow the approach of [7], where the representations mentioned above were investigated for real hyperbolic spaces. The main instrument here is the restriction to a maximal compact subgroup K = S(U(p) × U(q)). We determine K-types, find zonal spherical functions for these K-types, study the structure of representations associated with a cone (irreducibility, composition series etc.), determine intertwining operators and invariant Hermitian forms (both in matrix and integral form), indicate which representations (including those on subfactors) are unitarizable.

This paper is a development of a note by the author [8] which was published in Russian in a collection difficult to access.

The study of representations associated with a cone (understood in a wider sense: induced by arbitrary characters of a parabolic subgroup) for the group SU(1, q) and its universal covering group is undertaken in [11]. This paper follows the approach of [10], where a description of maximal degenerate series representations of the universal covering of the group SU(n, n) was given.

1991 Mathematical Subject Classification. 22E46..

Key words and phrases. Hyperbolic spaces, isotropic cone, representations, composition series, intertwining operators, sesqui-linear forms, unitarizability.

The problem of describing of maximal degenerate representations for hyperbolic spaces was considered in [3], [4], [5] by different methods, in different form and with different argumentations.

1. The pseudo-unitary group and its subgroups

Let G denote the group SU(p, q) of complex matrices of order n = p + q with determinant 1 preserving the following Hermitian form in ℂ^{n}:

[x, y] = −x_{1}ȳ_{1} + ⋯ + x_{p}ȳ_{p} + x_{p+1}ȳ_{p+1} + ⋯ + x_{n}ȳ_{n}.

We assume p ≥ 2, q ≥ 2 (so that n ≥ 4). We shall write vectors of ℂ^{n} in row form, so that G acts on ℂ^{n} from the right: x ↦ xg. Thus we have

ḡ′Ig = I,

where I = diag {−E_{p}, E_{q}}, E_{r} is the unit matrix of order r, the prime denotes matrix transposition.

Let us write g ∈ G in block form with respect to the partition n = p + q:

g = ( \begin{matrix}α & β \\ γ & δ \end{matrix} )  (1.1)

The Lie algebra 𝔤 of the group G consists of complex zero trace matrices X of order n satisfying the condition

X̄′I + IX = 0.

A matrix X in 𝔤 can be written in block form (as (1.1)):

X = ( \begin{matrix}iξ & η \\ η̄′ & iζ \end{matrix} ) ,  (1.2)

where ξ and ζ are Hermitian matrices, tr ξ + tr ζ = 0.

Let us consider the two following automorphisms θ and σ of G:

θ(g) = IgI, σ(g) = JgJ,

where J = diag {1, …, 1, −1}. They commute with each other. Notice that both are inner. For the corresponding automorphisms of the Lie algebra 𝔤, we preserve the notation θ and σ. Let K and H denote the fixed point subgroups of θ and σ, and 𝔨 and 𝔥 their Lie algebras respectively. The Lie algebra 𝔤 decomposes into the direct sums of +1, −1-eigenspaces of θ and σ respectively:

𝔤 = 𝔨 + 𝔭 = 𝔥 + 𝔮.

The Lie algebra 𝔨 consists of block diagonal matrices (1.2) with η = 0. It has one-dimensional centre spanned by

X_{0} = \frac{1}{n} ( \begin{matrix}−iqE_{p} & 0 \\ 0 & ipE_{q} \end{matrix} ) .  (1.3)

The semisimple part 𝔨^{(s)} = [𝔨, 𝔨] consists of matrices (1.2) with η = 0 and tr ξ = 0, tr ζ = 0. We have the decomposition:

𝔨 = ℝX_{0} + 𝔨^{(s)}.

The subspace 𝔭 consists of matrices (1.2) with ξ = ζ = 0.

An analytic subgroup of G with the Lie algebra 𝔨 is the subgroup K consisting of block diagonal matrices

( \begin{matrix}φ & 0 \\ 0 & ψ \end{matrix} ) ,  (1.4)

where φ ∈ U(p), ψ ∈ U(q), and detφ detψ = 1, so that K = S(U(p) × U(q)). It is a maximal compact subgroup of G. The space G/K is a Hermitian symmetric space. Let K^{(s)} be the subgroup of K with the Lie algebra 𝔨^{(s)}.

It consists of matrices (1.4) with detφ = 1, detψ = 1, so that K^{(s)} = SU(p) × SU(q).

Similarly the subgroup H consists of block diagonal matrices with respect to the partition n = (n − 1) + 1. It is isomorphic to U(p, q − 1). Its Lie algebra 𝔥 has one-dimensional centre with a basis diag {i, …, i, −(n − 1)i}. The space G/H is called a complex hyperbolic space (see, for example, [14], [9], [3] in connection with that). It has dimension 2n − 2 and rank 1. The latter means that Cartan subspaces in 𝔮 have dimension 1. As this one let us take the subspace 𝔞 of 𝔮 with a basis:

A_{0} = ( \begin{matrix}0 & 0 & 1 \\ 0 & 0 & 0 \\ 1 & 0 & 0 \end{matrix} ) ,  (1.5)

Here and further a matrix 3 × 3 denotes a matrix n × n written in the block form according to the partition n = 1 + (n − 2) + 1.

The Lie algebra 𝔤 decomposes into the direct sum of root spaces of the pair (𝔤, 𝔞):

𝔤 = 𝔤_{−2α} + 𝔤_{−α} + 𝔤_{0} + 𝔤_{α} + 𝔤_{2α}.

The subspace 𝔤_{jα} is an eigenspace of adA_{0} with the eigenvalue j. The subspace 𝔤_{0} is the direct sum 𝔪 + 𝔞 where the subalgebra 𝔪 consists of matrices

( \begin{matrix}iu & 0 & 0 \\ 0 & v & 0 \\ 0 & 0 & iu \end{matrix} ) ,  (1.6)

with u ∈ ℝ, , v ∈ 𝔲(p − 1, q − 1), 2u + tr v = 0. In its turn, 𝔪 decomposes into the direct sum:

𝔪 = 𝔪_{0} + 𝔪_{1},

of its one-dimensional centre 𝔪_{1} spanned by

Y_{0} = ( \begin{matrix}i & 0 & 0 \\ 0 & (−2i/(n − 2))E_{n−2} & 0 \\ 0 & 0 & i \end{matrix} ) ,  (1.7)

and the subalgebra 𝔪_{0} consisting of matrices (1.6) with u = 0.

The nilpotent subalgebra 𝔫 = 𝔤_{−α} + 𝔤_{−2α} corresponding to negative roots consists of matrices:

( \begin{matrix}it & ξ & it \\ ξ^{∗} & 0 & ξ^{∗} \\ −it & −ξ & −it \end{matrix} ) ,

where t ∈ ℝ, ξ is a row (ξ_{2}, …, ξ_{n−1}) in ℂ^{n}, ξ^{∗} is the column (−ξ̄_{2}, …, −ξ̄_{p}, ξ̄_{p+1}, …, ξ̄_{n−1}). Let A, M, M_{0}, M_{1}, N denote the analytic subgroups of G with the Lie algebras 𝔞, 𝔪, 𝔪_{0}, 𝔪_{1}, 𝔫 respectively. We have N = exp 𝔫, M_{1} = exp ℝY_{0}. The subgroup M consists of matrices

m = ( \begin{matrix}e^{iu} & 0 & 0 \\ 0 & w & 0 \\ 0 & 0 & e^{iu} \end{matrix} ) ,  (1.8)

where w ∈ U(p − 1, q − 1). It is the centralizer of 𝔞 in H. The subgroup M_{0} consist of matrices (1.8) with u = 0 and w ∈ SU(p − 1, q − 1).

There is an Iwasawa decomposition:

G = MANK,  (1.9)

i.e. any g ∈ G can be written as g = mank, m ∈ M, a ∈ A, n ∈ N, k ∈ K. This decomposition is defined up to elements in K∩M: g = mank = m_{1}a_{1}n_{1}k_{1} where m_{1} = ml, n_{1} = l^{−1}nl, k_{1} = l^{−1}k, l ∈ K ∩ M. In particular, the factor m can be taken in M_{0} (since M_{1} ⊂ K). Hence instead of (1.9) we can write another Iwasawa decomposition

G = M_{0}ANK.  (1.10)

The factor a for both decompositions is defined uniquely. For an explicit expression, see (2.4) below. It follows from (1.10) that

G/M_{0}AN = K/K ∩ M_{0}.

2. The manifold S

Let Y denote the cone [y, y] = 0, y ≠ 0. The group G acts on Y transitively by translations: y ↦ yg. Let us introduce the following Hermitian form on ℂ^{n}:

⟨x, y⟩ = x_{1}ȳ_{1} + … + x_{p}ȳ_{p}  (2.1)

and denote

|x| = \sqrt{⟨x, x⟩}.

Let S be the submanifold of Y defined by the equation |y| = 1. It is the direct product of two unit spheres: S = S_{1} × S_{2} where S_{1} = S^{2p−1} and S_{2} = S^{2q−1}. The group G acts on S as follows:

s ↦ s̃ = s · g = \frac{sg}{|sg|}.  (2.2)

In particular, the group K acts on S transitively by translations (rotations): s ↦ s · k = sk. Let us take as a basic point of S the point

s^{0} = (1, 0, …, 0, 1).

We have

s^{0}a = e^{t}s^{0}, s^{0}m = e^{iu}s^{0},

where a = exp tA_{0} and m has the form (1.8). Therefore, for g = mank (the Iwasawa decomposition (1.9)) we have

s^{0}g = e^{iu}e^{t}s^{0}k,  (2.3)

so that the stabilizer of s^{0} in G with respect to translations is M_{0}N. For the action (2.2), the stabilizer of s^{0} in G is M_{0}AN, so that S = G/M_{0}AN and S = K/K ∩ M_{0}.

Equality (2.3) allows to determine the factor a = exp tA_{0} in the Iwasawa decomposition g = mank:

|s^{0}g| = e^{t}.  (2.4)

Since M_{1} commutes with M_{0}, it acts on S from the left:

s^{0}k ↦ m_{1}s^{0}k = s^{0}m_{1}k,

so that in fact it is the multiplication of a vector s = s^{0}k by a number: if m_{1} = exp uY_{0}, see (1.7), then

m_{1}s = e^{iu}s.

Let us take on S the Euclidean measure ds:

ds = du dv,

where du, dv are the Euclidean measures on the unit spheres S_{1}, S_{2} respectively. The volume of the whole S is equal to

vol S = Ω_{2p} Ω_{2q} = \frac{4π^{n}}{Γ(p)Γ(q)},

recall that the volume of the unit sphere in ℝ^{m} is equal to

Ω_{m} = 2π^{m/2}/ Γ (\frac{m}{2}) .

Under the action (2.2) the measure ds is transformed as follows:

ds̃ = |sg|^{2−2n} ds.  (2.5)

In particular, this measure is invariant with respect to K.

Let (ψ, φ) denote the L^{2} inner product with respect to ds:

(f_{1}, f_{2}) = ∫_{S} f_{1}(s)\overline{f_{2}(s)}ds.  (2.6)

3. Representations associated with a cone

Let σ ∈ ℂ, τ ∈ ℤ. Define the following character (an one-dimensional representation) of the parabolic subgroup MAN:

ω_{σ,τ}(man) = e^{iτu}e^{σt}

where m has the form (1.8), a = exp tA_{0}. Let T_{σ,τ} be the representation of G induced by the character ω_{σ,τ} of MAN. It acts on the space D_{σ,τ}(G) of functions φ in C^{∞}(G) satisfying the condition

f(mang) = ω_{σ,τ}(man)f(g)  (3.1)

by right translations:

(T_{σ,τ}(g)f)(g_{1}) = f(g_{1}g)

Let us realize this representation on functions on S. By the Iwasawa decomposition (1.9), a function f ∈ D_{σ,τ}(G) is defined by its restriction ψ to K. In virtue of (3.1), ψ satisfies the conditions

ψ(l_{0}k) = ψ(k),  (3.2)
ψ(m_{1}k) = e^{iτu}ψ(k),  (3.3)

where l_{0} ∈ K ∩ M_{0}, m_{1} = exp uY_{0}. By (3.2), ψ defines a function φ on S = K/K ∩ M_{0}:

φ(s^{0}k) = ψ(k)

and (3.3) gives that φ satisfies the condition

φ(λs) = λ^{τ}φ(s) (|λ| = 1)  (3.4)

Thus, D_{σ,τ}(G) can be identified with the space D_{τ}(S) consist of C^{∞} functions φ on S with condition (3.4). In this realization the representation T_{σ,τ} looks as follows ( for s · g, see (2.2)):

(T_{σ,τ}(g)φ)(s) = φ(s · g)|sg|^{σ}

Indeed, taking s = s^{0}k, decompose kg by the Iwasawa decomposition (1.10): kg = m̃_{0}ãñk̃, then sg = s^{0}kg = s^{0}m̃_{0}ãñk̃ = et̃s^{0}k̃, so that |sg| = et̃ and s · g = s^{0}k̃.

A topology in D_{τ}(S) is introduced from D(S) (for a manifold M, D(M) denotes the space of compactly supported complex-valued C^{∞}-functions endowed by the Schwartz topology). The representations T_{σ,τ} are continuous, in the sense that the function T_{σ,τ}(g)φ from G × D_{τ}(S) into D_{τ}(S) is continuous, and indefinitely differentiable, i.e., for each φ ∈ D_{τ}(S) the function T_{σ,τ}(g)φ from G to D_{τ}(S) is of class C^{∞}.

The Hermitian form (2.6) is well-defined on D_{τ}(S) in virtue of (3.4). It is invariant with respect to the pair (T_{σ,τ}, T_{σ̄^{∗},τ}), i.e.,

(T_{σ,τ}(g)ψ, T_{σ̄^{∗},τ}(g)φ) = (ψ, φ)  (3.5)

where

σ^{∗} = 2 − 2n − σ  (3.6)

(formula (3.5) is proved by (2.5)).

The representation T_{σ,τ} of G gives rise to a representation of the Lie algebra 𝔤 which we denote by the same symbol. For X ∈ 𝔤 we have

(T_{σ,τ}(X)φ)(s) = (L_{X}φ)(s) + σRe⟨sX, s⟩

where L_{X} is the Lie operator of the action (2.2):

(L_{X} φ) (s) = \frac{d}{dt}|_{t=0} φ(s · exp tX), X ∈ 𝔤

and ⟨·, ·⟩ is given by (2.1).

The restriction of T_{σ,τ} to K is the representation R_{τ} of K by rotations on D_{τ}(S):

(R_{τ}(k)φ)(s) = φ(sk)  (3.7)

4. Harmonic analysis of the manifold S

Let R be the representation of the group K = S(U(p) × U(q)) on D(S) by rotations:

R(k)φ(s) = φ(sk).

It is unitary with respect to (2.6). It splits into the representations R_{τ} on D_{τ}(S), τ ∈ ℤ, see (3.7). In this Section we decompose every R_{τ} into irreducible constituents.

First consider the unit sphere S_{1} = S^{2p−1}, see § 2. It consists of points (s_{1}, …, s_{p}) ∈ ℂ^{p}, satisfying the condition s_{1}s̄_{1} + … + s_{p}s̄_{p} = 1. As it is known [12], the space D(S_{1}) decomposes into the direct sum of the subspaces H(l), l ∈ ℕ = {0, 1, 2, … } (recall p > 1). The subspace H(l) consists of the restrictions to S_{1} of homogeneous harmonic polynomials in x_{1}, x̄_{1}, …, x_{p}, x̄_{p} of degree l with complex coefficients. Denote by the same symbol H(l) the space of those polynomials. Let P(r), A(r), Ā(r) denote the spaces of all, analytic and antianalytic homogeneous polynomials of degree r respectively. Here is an expression for the dimension:

dim A(r) = dim Ā(r) = \frac{(r + p − 1)!}{r!(p − 1)!}.  (4.1)

Under the natural action of U(p) the spaces A(r) and Ā(r) are irreducible, and the space H(l) decomposes into the sum of the subspaces

H(l, u) = [A(l − u) ⊗ Ā(u)] ∩ H(l),  (4.2)

where u = 0, 1, …, l. Let us denote by D(r), D̄(r), D(l, u) the corresponding representations of U(p) on A(r), Ā(r), H(l, u) respectively.

Lemma 4.1. The representations D(l, u) of the group U(p) are irreducible and pairwise non-equivalent. The dimension of H(l, u) is equal to

(l + p − 1) \frac{(l − u + p − 2)! (u + p − 2)!}{(l − u)! u! (p − 2)! (p − 1)!}  (4.3)

Proof. The highest weights of D(r) and D̄(r) are (r, 0, …, 0), (0, …, 0, −r), respectively (p coordinates). The highest component D′(l, u) of D(l−u)⊗D̄(u) has the highest weight

(l − u, 0, …, 0, −u).  (4.4)

By the Weyl formula (see, for example, [15] § 73), the dimension of D′(l, u) is given by (4.3).

On the other hand [12],

P(l) = H(l) + Q · H(l − 2),

where Q = x_{1}x̄_{1} + … + x_{p}x̄_{p}. Therefore,

A(l − u) ⊗ Ā(u) = H(l, u) + Q [A(l − u − 1) ⊗ Ā(u − 1)].  (4.5)

The highest weight of the second term in the right hand side of (4.5) is (l−u−1, 0, …, 0, −u + 1), it is lower than (4.4). Therefore, D′(l, u) ⊂ D(l, u). By (4.5) the dimension of H(l, u) is equal to the remainder:

dim A(l − u) dim Ā(u) − dim A(l − u − 1) dim Ā(u − 1).

Substitute here (4.1). A computation shows that dim H(l, u) is precisely (4.3). Thus, the dimensions of both representations D′(l, u) and D(l, u) coincide, so that these representations coincide too. Therefore, D(l, u) is irreducible. The non-equivalence follows from (4.4). □

Lemma 4.2. The restriction of the representation D(l, u) to the group SU(p) is irreducible. For p > 2, these representations of SU(p) are pairwise non-equivalent. For q = 2, the restrictions with the same l are equivalent (they are equivalent to the representation with the highest weight (l, 0)).

Indeed, the restriction of D(l, u) to SU(p) has the highest weight (l, u, …, u, 0). □

Lemma 4.3. The space of polynomials in H(l, u) depending only on x_{1}, x̄_{1}, Q is one-dimensional and is spanned by the following polynomial:

f = x_{1}^{l−2u} Q^{u}F (−u, l − u + p − 1; l − 2u + 1; x_{1}x̄_{1}/Q), l ≥ 2u,  (4.6)
f = x̄_{1}^{2u−l} Q^{l−u}F (u − l, u + p − 1; 2u − l + 1; x_{1}x̄_{1}/Q), l ≤ 2u,  (4.7)

where F is the Gauss hypergeometric function ([2], Ch. 2).

Proof. Let l ≥ 2u. Then by (4.2) a polynomial f ∈ H(l, u) depending on x_{1}, x̄_{1}, Q has the following form:

f = x_{1}^{l−2u}g(a, Q),

where a = x_{1}x̄_{1} and g is a homogeneous polynomial in two variables of degree u. By the homogeneity, f can be written as follows:

f = x_{1}^{l−2u}Q^{u}h(a/Q),  (4.8)

where h(z) = g(z, 1) is a polynomial in one variable of degree u. Further, f satisfies the Laplace equation:

∑_{i=1}^{p} \frac{∂^{2}f}{∂x_{i}∂x̄_{i}} = 0.

This equation gives an equation for the polynomial h(z) from (4.8):

z(1 − z)\frac{d^{2}h}{dz^{2}} + [l − 2u + 1 − (l − 2u + p)z]\frac{dh}{dz} + u(l − u + p − 1)h = 0.

It is a hypergeometric equation. Its solution regular at zero is:

F (−u, l − u + p − 1; l − 2u + 1; z).

It gives (4.6). Similarly we consider the case l ≤ 2u and obtain (4.7). □

Let us denote the space of restrictions to S_{1} of polynomials in H(l, u) by the same symbol H(l, u). The following lemma follows immediately from Lemma 4.3.

Lemma 4.4. The space of functions on S_{1} in H(l, u) depending only on s_{1} is one-dimensional. The basis function normalized by 1 at the point (1, 0, …, 0) is ψ(p, l, u; s_{1}) where for l ≥ 2u:

ψ(p, l, u; t) = c(l, u) t^{l−2u} F (−u, l − u + p − 1; l − 2u + 1; tt̄),  (4.9)
c(l, u) = (−1)^{u} \frac{Γ(l − u + 1) Γ(p − 1)}{Γ(l − 2u + 1) Γ(u + p − 1)},  (4.10)

and F is the Gauss hypergeometric function; for l ≤ 2u:

ψ(p, l, u; t) = ψ(p, l, l − u; t̄)

Formula (4.9) can be rewritten in terms of the Jacobi polynomials, see [2] 10.8 (16):

ψ(p, l, u; t) = (−1)^{u} \frac{u!(p − 2)!}{(u + p − 2)!} P_{u}^{(l−2u, p−2)}(1 − 2tt̄), l ≥ 2u.

Let us go to S = S_{1} × S_{2}. We see that the space D(S) decomposes into the sum of the spaces H(ζ) = H(l, u) ⊗ H(m, v) where ζ denotes the quadruple (l, m; u, v), l, m, u, v ∈ ℕ with u ≤ l, v ≤ m. By Lemma 4.2 they are irreducible under K^{(s)} = SU(p) × SU(q), therefore, under K. Let us denote by R(ζ) the representation of K on H(ζ) by rotations.

The following two theorems follow from Lemmae 4.2, 4.4.

Theorem 4.5. The representation R of K on D(S) by rotations decomposes into the direct sum of the irreducible representations R(ζ). If p > 2, q > 2, then these representations R(ζ) are pairwise non-equivalent.

Theorema 4.6. For each H(ζ), ζ = (l, m; u, v), the subspace of functions depending only on s_{1}, s_{n} is one-dimensional, the basis function ψ_{ζ} normalized by 1 at s^{0} is

ψ_{ζ}(s) = ψ(p, l, u; s_{1})ψ(q, m, v; s_{n}).

Now we get the decomposition of the space D_{τ}(S): it is the direct sum of H(ζ) with ζ satisfying the condition:

l + m − 2u − 2v = τ.  (4.11)

It follows from (4.11) that

l + m ≡ τ (mod 2)  (4.12)

and

l + m ≥ |τ|.  (4.13)

Theorem 4.7. The representation R_{τ} of K on D_{τ}(S) by rotations decomposes into the direct sum of the irreducible representations R(ζ) with ζ = (l, m; u, v) satisfying (4.18). These representations R(ζ) are pairwise non-equivalent.

Proof. The only thing to prove is the non-equivalence for p = 2 or q = 2. In the representation R(ζ) the basis element X_{0} of the centre of 𝔨, see (1.3), is the multiplication by a number:

R(ζ)(X_{0}) = λ(ζ)id,

where

λ(ζ) = \frac{i}{n}(−(l − 2u)q + (m − 2v)p) =

= i(2u − l) + iτp/n.

These numbers are different for the representations R(ζ) with the same l and different u. Therefore, these representations are non-equivalent. It proves our statement for p = 2. Similarly we consider the case q = 2. □

The functions ψ_{ζ} are pairwise orthogonal with respect to (2.6) and, see [7],

(ψ_{ζ}, ψ_{ζ}) = vol S/H(ζ).

5. The structure of representations associated with a cone

From now on we restrict ourselves to the representations T_{σ,τ} with τ = 0. Just these representations occur in the decomposition of the quasiregular representation on the complex hyperbolic spaces. So we either omit the index τ and write T_{σ} instead of T_{σ,τ} etc. or replace τ by 0, for example, R_{0}. The space D_{0}(S) will be denoted be V. The conditions (4.11) and (4.12) become

l + m − 2u − 2v = 0,  (5.1)
l + m ≡ 0 (mod 2)  (5.2)

and (4.13) becomes trivial.

In this Section we study the structure of the representations T_{σ} (irreducibility, composition series etc.). We use the restriction to K. This restriction is the representation R_{0} of K which was studied in §4, see Theorem 4.6 and Theorem 4.7.

Now let us consider the action of the element A_{0}, see (1.5), in the representation T_{σ}. Since A_{0} commutes with M, the operator T_{σ}(A_{0}) preserves the space functions depending on s_{1} and s_{n} only, so that a function ψ_{ζ} goes to a linear combination of the same functions. Before to formulate the result, we need to do some preparations (cf. [7]).

Denote by z the pair (l, m), l, m ∈ ℕ, and call it a weight. Let ⟨z_{1}, z_{2}⟩ be the standard inner product: ⟨z_{1}, z_{2}⟩ = l_{1}l_{2} + m_{1}m_{2}. Let Z denote the set of weight z satisfying (5.2). Introduce the following 4 vectors on the plane ℝ^{2}:

e_{1} = (1, 1), e_{2} = (1, −1), e_{3} = (−1, 1), e_{4} = (−1, −1).

and the following 4 linear functions β_{i} of z = (l, m):

β_{1}(σ; z) = σ − l − m,  (5.3)
β_{2}(σ; z) = σ − l + m + 2q − 2,  (5.4)

β_{3}(σ; z) = σ + l − m + 2p − 2,  (5.5)
β_{4}(σ; z) = σ + l + m + 2n − 4.  (5.6)

For i = 1, 2, 3, 4, we denote i′ = 5 − i, so that e_{i′} = −e_{i}. Notice that

β_{i}(σ; z) + β_{i′}(σ^{∗}; z) = −2

A quadruple ζ = (l, m; u, v) ∈ ℝ^{4}, see §4, can be written as ζ = (z; u, v), where z = (l, m). In this case we shall say that ζ is subordinate to z. Denote by H(z) the direct sum of H(ζ) with ζ subordinate to z:

H(z) = ∑_{ζ=(z;u,v)} H(ζ),

Notice that if ζ = (z; u, v) then

ζ + e_{ij} + e_{i′j} = (z; u ± 1, v ∓ 1),  (5.7)
ζ + e_{ij} + e_{i′j′} = ζ.  (5.8)

Further, introduce 8 vectors on ℝ^{4} = ℝ^{2} ⊕ ℝ^{2}:

e_{ij} = e_{i} ⊕ (1/2)(e_{i} + e_{j}),

where i = 1, 2, 3, 4, j = 2, 3.

Theorem 5.1.

T_{σ}(A_{0})ψ_{ζ} = ∑_{i,j} c_{ij}(ζ)β_{i}(σ; z)ψ_{ζ+e_{ij}}  (5.9)

where c_{ij}(ζ) are given

c_{ij}(ζ) = \frac{a_{ij}(p, l, u)a_{<i>,<j>}(q, m, v)}{2(l + p − 1)(m + q − 1)},

the functions a_{ij} = a_{ij}(p, l, u) are the following:

a_{12} = a_{22} = u + p − 1, a_{32} = a_{42} = l − u,
a_{13} = a_{23} = l − u + p − 1, a_{33} = a_{43} = u,

and the correspondence i ↦< i > is the substitution (1234) ↦ (1324).

Proof. The proof is rather long and cumbersome computations with hypergeometric functions. So we restrict ourselves here to an outline of the demonstration for one case: l > 2u. Then by (5.1) we have m < 2v.

For simplicity, we rename: s_{1} = x, s_{n} = y. On functions depending on these variables only, the operator T_{σ}(A_{0}) is:

T_{σ}(A_{0}) = σN + L,  (5.10)

where

N = Re xȳ = (1/2)(xȳ + x̄y),

L = y \frac{∂}{∂x} + x\frac{∂}{∂y} + ȳ\frac{∂}{∂x̄} + x̄\frac{∂}{∂ȳ} − x\frac{∂}{∂x} − y \frac{∂}{∂y} + x̄\frac{∂}{∂x̄} + ȳ\frac{∂}{∂ȳ}.

Now it is more convenient for us to take a function in H(ζ) depending on s_{1} = x, s_{n} = y without factors c, see (4.9), (4.10), namely,

φ_{ζ} = f_{l,u}(x)g_{m,v}(y)  (5.11)

where

f_{l,u}(x) = x^{l−2u} F (−u, l − u + p − 1; l − 2u + 1; a),  (5.12)
g_{m,v}(y) = ȳ^{2v−m} F (−m + v, v + q − 1; 2v − m + 1; b),  (5.13)
a = xx̄, b = yȳ,

so that

ψ(p, l, u; x) = c(l, u)f_{l,u}(x), ψ(q, m, v; y) = c(m, m − v)g_{m,v}(y). (5.14)

For brevity, denote the hypergeometric functions in (5.12), (5.13) by F_{1}, F_{2} respectively. Denote

μ = l − 2u, ν = 2v − m.

Apply the operator (5.10) to (5.11). We obtain

T_{σ}(A_{0})f_{lu}g_{mv} = (σ −μ−ν)·N ·f_{lu}g_{mv} +μx^{μ−1}F_{1} ·ȳ^{ν−1}bF_{2} +x^{μ−1}aF_{1} ·νȳ^{ν−1}F_{2}+

+x^{μ}ȳ^{ν}(xȳ + x̄y){(1 − a)F′_{1}F_{2} + (1 − b)F_{1}F′_{2}},

where the prime denotes the differentiation, or

T_{σ}(A_{0})f_{lu}g_{mv} = \frac{1}{2}(σ − μ − ν)x^{μ+1}F_{1} · ȳ^{ν+1}F_{2}+

+\frac{1}{2}(σ − μ − ν)x^{μ−1}aF_{1} · ȳ^{ν−1}bF_{2}+

+ [μx^{μ−1}F_{1} + x^{μ−1}a(1 − a)F_{1}] ȳ^{ν−1}bF_{2}+

+x^{μ−1}aF_{1} [νȳ^{ν−1}F_{2} + ȳ^{ν−1}b(1 − b)F_{2}] +

+x^{μ+1}(1 − a)F′_{1} · ȳ^{ν+1}F_{2} + x^{μ+1}F_{1} · ȳ^{ν+1}(1 − b)F′_{2}  (5.15)

Let us take the following notation system. For hypergeometric function F (α, β; γ; z), we denote, for example, by F (+0, +1, +1) the hypergeometric function which is obtained from the given function if we add numbers 0, +1, +1 to the parameters α, β, γ respectively. Side by side with f_{l,u}(x), g_{m,v}(y), let us consider the “adjacent” functions:

f_{l+1,u} = x^{μ+1}F_{1}(+0, +1, +1)
f_{l+1,u+1} = x^{μ−1}F_{1}(−1, +0, −1)
f_{l−1,u} = x^{μ−1}F_{1}(+0, −1, −1)
f_{l−1,u−1} = x^{μ+1}F_{1}(+1, +0, +1)
g_{m+1,v+1} = ȳ^{ν+1}F_{2}(+0, +1, +1)
g_{m+1,v} = ȳ^{ν−1}F_{2}(−1, +0, −1)
g_{m−1,v} = ȳ^{ν+1}F_{2}(+1, +0, +1)
g_{m−1,v−1} = ȳ^{ν−1}F_{2}(+0, −1, −1)

We shall use some relations for hypergeometric functions F (α, β; γ; z). Let us write then in our notation.

F′ = \frac{αβ}{γ(β − α)(1 − z)} \{(γ − α)F (+0, +1, +1) − (γ − β)F (+1, +0, +1)\} ,

(5.16)

γ(β − α)F = α(β − γ)F (+1, +0, +1) + β(γ − α)F (+0, +1, +1),

(5.17)

zF = \frac{γ − 1}{β − α} \{F (+0, −1, −1) − F (−1, +0, −1)\} ,

(5.18)

(γ − 1)F + z(1 − z)F′ = \frac{γ − 1}{β − α} \{βF (+0, −1, −1) − αF (−1, +0, −1)\} .

(5.19)

Formula (5.16) follows from [2] 2.8(20) and 2.8(37), formula (5.17) is proved by means [2] 2.8(32), (38), (43), formula (5.18) can be proved by expanding in power series, formula (5.19) is proved by [2] 2.8(20), (32) and our formula (5.18).

Applying these relations to each factor in the terms in (5.15), we can express all these factors in terms of adjacent functions, so that we get

T_{σ}(A_{0})φ_{ζ} = ∑_{i,j} C_{ij}(ζ)β_{i}(σ; z)φ_{ζ+e_{ij}}

Now we return from φ_{ζ} to ψ_{ζ} by (5.14) and obtain (5.9). □

We say that the representation T_{σ} links ζ with ζ′ if there exist X ∈ 𝔤 and φ ∈ H(ζ) such that T_{σ}(X)φ has a non-zero projection to H(ζ′).

Lemma 5.2. Let ζ = (z; u, v). The representation T_{σ} links ζ with ζ′ if and only if ζ′ = ζ + e_{ij} for some i, j and β_{i}(σ; z) ≠ 0.

Proof. The part “if” follows from Theorem 5.1 immediately. Let us prove the part “only if”.

Since the subspace H(ζ) is invariant with respect to K, it is sufficient to consider X ∈ 𝔭. The space 𝔭 is invariant with respect to AdK. Let us complexify 𝔭. The corresponding representation P of K on 𝔭^{ℂ} is the direct sum of two irreducible representations R(e_{12}) and R(e_{13}). They are the restrictions to K of the representations D̄(1) ⊗ D(1) and D(1) ⊗ D̄(1) of U(p) × U(q). Recall that highest weights of the representations D(1) and D̄(1) of U(p) are (1, 0, …, 0) and (0, …, 0, −1) respectively. Let us take in 𝔭, H(ζ) and H(ζ′) bases {L_{r}}, {φ_{a}}, {φ′_{b}} respectively – in such a way that φ_{1} = ψ(ζ), φ′_{1} = ψ(ζ′). By the Wigner-Eckart theorem (see, for example, [13], [5])

(T_{σ}(L_{r})φ_{a}, φ′_{b}) = M · ⟨b|r, a⟩  (5.20)

where ⟨b|r, a⟩ is a Clebsch-Gordan coefficient of the tensor product P ⊗ R(ζ) and M is a reduced matrix element, the latter depends on σ, ζ, ζ′ and does not depend on r, a, b. The tensor product P ⊗ R(ζ) decomposes into the sum ∑ R(ζ + e_{ij}) + … where the dots mean representations which are not realized on functions on S. Together with (5.20) it gives that T_{σ} can link ζ with ζ + e_{ij} only. On the other hand, it follows from (5.9) that for σ ∉ 2ℤ the link actually takes place, so that ⟨1|1, 1⟩ ≠ 0. Therefore, if β_{i}(σ; z) = 0, then by (5.9) and (5.20) we have M = 0 for ζ′ = ζ + e_{ij}. It follows that the left hand side of (5.20) is equal to 0 for all r, a, b, so that T_{σ} does not link ζ with ζ + e_{ij} for β_{i}(σ; z) = 0. □

Let us call the line β_{i}(σ; z) = 0 on the plane of z = (l, m) a barrier for T_{σ} if this line and the set {β_{i}(σ; z) < 0} have non-empty intersections with Z. The barrier β_{i} = 0 divides Z into two parts: the set β_{i} ≥ 0 (the interior of the barrier) and the set β_{i} < 0 (the exterior of the barrier). For brevity, sometimes we shall speak “the barrier i” instead of “the barrier β_{i} = 0”.

If β_{i} = 0 is a barrier, then we denote by V_{i} = V_{σ,i} the subspace of V which is the direct sum of H(z) with β_{i}(σ; z) ≥ 0. For brevity, we shall write V_{i}/V_{j} instead of V_{i}/(V_{i} ∩ V_{j}).

Theorem 5.3 The subspaces V_{σ,i} are invariant with respect to the representation T_{σ}. Any irreducible subfactor of T_{σ} is obtained by these subspaces (i.e. any irreducible subfactor is V, or V_{i}, or V_{i} ∩ V_{j}, factorized over a sum of several V_{k}). If σ ∉ 2ℤ, then T_{σ} is irreducible.

Proof. The invariance of V_{i} follows from Lemma 5.2 immediately. Now let U be an invariant subspace of V. Let Z(U) ⊂ Z be the set of z such that H(ζ) ⊂ U for some ζ subordinate to z. It just follows from Lemma 5.2, that Z(U) is a common interior part of several barriers. It remains to prove that if z = (l, m) ∈ Z(U) then U contains all H(ζ) with ζ subordinate to z. It is trivial if l = 0 or m = 0. Let H(ζ) ⊂ U, l, m ≠ 0. Then there exists i = 1, 2, 3, 4 such that β_{i}(σ; z) ≠ 0 and β_{i′}(σ; z + e_{i}) ≠ 0. Let us apply T_{σ}(X_{0}) first to H(ζ) and then to H(ζ + e_{i′,j}). By (5.9) we see that together with H(ζ) the space U contains H(ζ +e_{ij} +e_{i_{′},j}) = H(z; u±1, v ∓1), see (5.7). Therefore, U contains all H(ζ) with given z. At last, if σ ∉ 2ℤ, then barriers are absent. □

In order to see the structure of invariant subspaces, irreducible subfactors, composition series etc., it is sufficient to draw the barriers on quadrant l ≥ 0, m ≥ 0 for each σ and to endow each barrier by a bristle (dashes) oriented inside of the barrier (where β_{i} ≥ 0).

If σ = 1 − n, n odd, then V splits into the direct sum:

V = V_{2} + V_{3}  (5.21)

The irreducible subspace V_{2} ∩ V_{3} for σ = 2 − n, n even, has weights z lying on one ray, i.e. on coinciding barriers 2, 3. Similarly it goes for σ = −n, n even, for the subfactor V/(V_{2} + V_{3}). Let us call such subfactors exceptional.

The subfactors V_{1} and V/V_{4} are finite-dimensional. In particular, for σ = 0 and σ^{∗} = 0 they realize the unit representation of G.

For the subfactor W, let Z(W) denote the set of weights z ∈ Z occuring in W. For an invariant subspace U under T_{σ}, denote by U^{⊥} its orthogonal complement with respect to form (2.6). Then Z(U^{⊥}) = Z \ Z(U) and U^{⊥} is invariant under T_{}_{σ̄^{∗}}. For a subfactor W = U/Q of T_{σ} the subspace W^{∗} = Q^{⊥}/U^{⊥} is a subfactor of T_{}_{σ̄^{∗}}. We have Z(W^{∗}) = Z(W). Let us call W^{∗} the dual subfactor. The dual subfactors for V_{i} and V/V_{i} are V/V_{i′} and V_{i′} respectively.

6. Intertwining operators

A continuous operator A on V is said to intertwine the representations T_{σ} and T_{σ_{1}} if for any g ∈ G we have

T_{σ_{1}}(g) A = A T_{σ}(g).  (6.1)

Therefore, for X ∈ 𝔤 we have

T_{σ_{1}}(X) A = A T_{σ}(X).  (6.2)

A similar definition is given for the case when an operator A maps a subfactor W of T_{σ} in a subfactor W_{1} of T_{σ_{1}}.

Theorem 6.1. A non-zero intertwining operator A as above exists only in the following cases:

(a) σ_{1} = σ;
(b) σ_{1} = σ^{∗}.

In the case (a) such an operator is a scalar operator (the multiplication by a number) on H(ζ) except the case σ = 1 − n, n odd, then A is a scalar operator on each term in (5.21).

In the case (b) for any irreducible subfactor W of T_{σ} there exists a unique up to the factor non-zero operator mapping W onto the dual subfactor W^{∗} and intertwining the subfactor of T_{σ} on W with the subfactor of T_{σ^{∗}} on W^{∗}.

Proof. Let A be a continuous operator on V satisfying (6.1). It follows from Theorem 4.7, that A preserves each H(ζ) and its restriction to H(ζ) is the multiplication by a number a(ζ). In particular, we have

Aψ_{ζ} = a(ζ)ψ_{ζ}.

The numbers a(ζ) depend on σ, σ_{1}.

Take in (6.2) X = A_{0}, see (1.5), apply to ψ_{ζ} and use (5.9), then we obtain relations for a(ζ):

β_{i}(σ; z) a(ζ + e_{ij}) = β_{i}(σ_{1}; z) a(ζ),  (6.3)

These relations have to be co-ordinated, i.e. if one applies to a(ζ + e_{i}) the same relation with i, j replaced by i′, j′, then one has to get a(ζ) again, see (5.8). It gives the conditions:

β_{i}(σ; z + e_{i′}) β_{i′}(σ; z) a(ζ) = β_{i}(σ_{1}; z + e_{i′}) β_{i′}(σ_{1}; z) a(ζ).  (6.4)

Since A ≠ 0, there exists at least one ζ such that a(ζ) ≠ 0. Omitting this a(ζ) from (6.4) and substituting the explicit expressions (5.3)–(5.6), we obtain that a square trinomial (in σ) whose roots give the sum 2 − 2n has the same values at σ and σ_{1}. Therefore, we have σ_{1} = σ or σ_{1} = σ^{∗} (= 2 − 2n − σ), see (3.6).

First let σ_{1} = σ. The relations (6.3) show that a(ζ + e_{ij}) = a(ζ) for that ζ = (z; u, v) and i, j for which β_{i}(σ; z) ≠ 0. For any z, there exists i such that β_{i}(σ; z) ≠ 0 and β_{i′}(σ; z + e_{i}) ≠ 0. In virtue of (5.7), a(ζ) actually depends only on z to which ζ is subordinate. So let us write a(z) instead of a(ζ). The relations (6.3) give a(z + e_{i}) = a(z). This fact together with the same one for i′ gives that a(z) should be constant along any line z + te_{i} and, therefore, on the whole Z, except the case σ = 1 − n, n odd, when barriers 2,3 split Z into two non-linked subsets, so that there is the decomposition (5.21).

Now let σ_{1} = σ^{∗}. Then (6.3) is

β_{i}(σ; z) a(ζ + e_{ij}) = β_{i}(σ^{∗}; z) a(ζ).  (6.5)

Let us apply to a(ζ + e_{ij}) the same relation with i replaced by i′. We get that at least for one of i = 1, 2, 3, 4 there is the following equality:

a(ζ + e_{ij} + e_{i′j}) = a(ζ)

where j = 2, 3. By (5.7) we get again that a(ζ) actually depends only on z. So we write again a(z) instead of a(ζ). Thus,

Aφ = a(z)φ, φ ∈ H(z),

and the relations (6.5) become

β_{i}(σ; z) a(z + e_{i}) = β_{i}(σ^{∗}; z) a(z).  (6.6)

Let W = U/Q be some irreducible subfactor of T_{σ}. Similarly to [7] we can show that equations (6.6) have a unique up to the factor solution on the set Z(W) which is the intersection of the interiors of some barriers and, moreover, a(z) = 0 on Z(U) ∩ Z(Q). Namely, any solution of (6.6) on Z is:

a(z) = const · \frac{(−1)^{m}}{∏_{i=1}^{4} Γ(−\frac{1}{2}β_{i}(σ; z))}.  (6.7)

In the reducible case it vanishes on the interiors of bariers. A solution of (6.6) on Z(W) is

a(z) = const · ε(z) · \frac{∏_{j} Γ(1 + \frac{1}{2}β_{j}(σ; z))}{∏_{i} Γ(−\frac{1}{2}β_{i}(σ; z))},

in the nominator the product is taken over indexes j such that Z(W) lies inside of barriers j and in the denominator – over the rest indexes, ε(z) = (−1)^{r(z)} where r(z) = m + ∑(1/2)⟨e_{j}, z⟩, where the sum is taken over j from the nominator. This solution a(z) defines an operator on U vanishing on U ∩ Q so that it maps U/Q onto Q^{⊥}/U^{⊥} and intertwines corresponding subfactors of T_{σ} and T_{σ^{∗}}. Similarly to [7] it is shown that this operator is continuous. □

Let us consider an integral operator A_{σ} on V which is defined by:

(A_{σ}φ) (s) = ∫_{S} |[s, t]|^{σ^{∗}}φ(t) dt

The integral converges absolutely for Reσ < 4 − 2n and is extended to the σ-plane as a meromorphic function.

Theorem 6.2. The operator A_{σ} intertwines T_{σ} and T_{σ^{∗}}:

T_{σ^{∗}}(g) A_{σ} = A_{σ} T_{σ}(g)  (6.8)

On the subspace H(z), z = (l, m), it is the multiplication by a number:

A_{σ}φ = a(σ; z)φ, φ ∈ H(z),  (6.9)

where

a(σ; z) = 4 (−1)^{m} π^{n} \frac{Γ(1 − n − σ)Γ^{2}((4 − 2n − σ)/2)}{∏_{i=1}^{4} Γ(−\frac{1}{2}β_{i}(σ; z))}.  (6.10)

Proof. The proof of (6.8) is an easy computation with using (2.5). Now (6.8) implies (6.9), see the proof of Theorem 6.1.

First we compute a(0) = a(σ; 0). Set in (6.9) z = 0 (so that ψ_{ζ} = 1) and take it at s = s^{0}. Let us for the moment write λ instead of σ^{∗}. Then

a(0) = ∫_{S} |[s^{0}, s]|^{λ} ds = ∫_{S} |s_{1} − s_{n}|^{λ} ds.  (6.11)

Denote s_{1} = z = x + iy, s_{n} = w = u + iv. Integrating (6.11) in other variables, we get

a(0) = Ω_{2p−2}Ω_{2q−2} ∫ |z − w|^{λ}(1 − zz̄)^{p−2}(1 − ww̄)^{q−2}dxdydudv

where the integral is taken over |z| ≤ 1, |w| ≤ 1. Let use a trick of Faraut [3]: we replace |z − w|^{λ} by

|z − w|^{λ} = B (\frac{1}{2}(λ + 1), \frac{1}{2})^{−1} ∫_{0}^{2π} |Re((z − w)e^{it})|^{λ}dt,

where B is the Euler beta function. Then

a(0) = Ω_{2p−2}Ω_{2q−2}πB (\frac{1}{2}(λ + 1), \frac{1}{2})^{−1} ·

· ∫ |x − u|^{λ}(1 − x^{2} − y^{2})^{p−2}(1 − u^{2} − v^{2})^{q−2}dxdydudv.

Integrating over y and v, we obtain

a(0) = Ω_{2p−2}Ω_{2q−2}πB (\frac{1}{2}(λ + 1), \frac{1}{2})^{−1} B (p − 1, \frac{1}{2}) B (q − 1, \frac{1}{2}) ·

· ∫ |x − u|^{λ}(1 − x^{2})^{p−3/2}(1 − u^{2})^{q−3/2}dxdu  (6.12)

where the integral is taken over |x| ≤ 1, |u| ≤ 1. The latter integral is a particular case of an integral found in [7] (5.7). It equals to

\frac{2^{2−λ}π^{n}}{Ω_{2p−1}Ω_{2q−1}} · \frac{Γ(λ + 1)Γ(λ + n − 1)}{∏_{i=1}^{4} Γ(−\frac{1}{2}β_{i}(λ^{∗}; 0))}  (6.13)

Substituting (6.13) in (6.12) instead of the integral and remembering λ = σ^{∗}, we find

a(0) = 4π^{n} \frac{Γ(1 − n − σ)Γ^{2}((4 − 2n − σ)/2)}{∏_{i=1}^{4} Γ(−\frac{1}{2}β_{i}(σ; 0))}.

This allows to find const in (6.7) and, as result, we have (6.10). □

7. Unitarity

A sesqui-linear form H(ψ, φ) on V is called invariant with respect to the pair T_{σ}, T_{σ_{1}}, if for any g ∈ G and any ψ, φ ∈ V we have

H(T_{σ}(g)ψ, T_{σ_{1}}(g)φ) = H(ψ, φ).

It is the same that

H(T_{σ}(X)ψ, φ) + H(ψ, T_{σ_{1}}(X)φ) = 0

for any X ∈ 𝔤. Similarly we define the invariance of a form on a pair of invariant subfactors W/U and W_{1}/U_{1}; then ψ ∈ W, φ ∈ W_{1} and H(ψ, φ) = 0 if ψ ∈ U or φ ∈ U_{1}.

Theorem 7.1. A non-zero sesqui-linear form H(ψ, φ) invariant with respect to the pair T_{σ}, T_{σ_{1}} on V or on a pair of invariant subfactors exists only in the following cases:

(a) σ_{1} = σ̄^{∗};
(b) σ_{1} = σ̄.

In the case (a) the form H coincides up to the factor with the form (2.6) except σ = 1 − n, n odd. In the latter case the form H coincides up to the factor with (2.6) on each of subspaces V_{2}, V_{3}.

In the case (b) the form H can be expressed in terms of the form (2.6) and the operator A intertwining T_{σ} and T_{σ^{∗}} or their subfactors, see Theorem 6.1, namely

H(ψ, φ) = (Aψ, φ).

Proof. By Theorem 4.7, the subspaces H(ζ) are pairvise orthogonal with respect to H.

Therefore, H has a “diagonal form”:

H(ψ, φ) = h(ζ) (ψ, φ)

if both ψ, φ belong to H(ζ) and H(ψ, φ) = 0 if ψ, φ belong to different H(ζ). Further the study of h(ζ) goes similarly to the study of a(ζ) in the proof of Theorem 5.1, so we omit it. □

In particular, h(ζ) actually depends only on z: h(ζ) = h(z), so that

H(ψ, φ) = h(z) (ψ, φ)  (7.1)

for ψ, φ ∈ H(z).

Now determine when representations T_{σ} are unitarizable. For that we have to set σ_{1} = σ in Theorem 7.1 and the form H(ψ, φ) has to be Hermitian (H(ψ, φ) real) and positively definite. For σ we obtain two possibilities: σ = σ̄^{∗} and σ = σ̄. In the first case (σ = σ̄^{∗}) we have Re σ = 1 − n. So we obtain a series of unitarizable representations T_{σ} of G, the invariant inner product is (2.6), so that the unitary completion acts on the space of functions ψ in L^{2}(S) satisfying φ(λs) = φ(s), |λ| = 1. Let us call this series the continuous series. Representations of this series are irreducible except the split case σ = 1 − n, n odd (when V = V_{2} + V_{3}).

Let us consider the second case: σ = σ̄, i.e. σ ∈ ℝ. First consider the irreducible case.

Theorem 7.2. An irreducible representation T_{σ} with σ ∈ ℝ can be unitarizable only for n even. These σ fill the interval (−n, 2−n). The inner product is, for example,

(−1)^{p−1} \frac{1}{Γ(σ + n − 1)}(A_{σ}ψ, φ).

For n odd, there is no irreducible unitarizable representation T_{σ} with σ ∈ ℝ.

Let us call the family of representations pointed out in this theorem the complementary series.

Proof. As well as a(z) in section 6, the factor h(z) from (7.1) satisfies the equation:

β_{i}(σ; z) h(z + e_{j}) = β_{i}(σ^{∗}; z) h(z).  (7.2)

So we have to learn when equations (7.2) have a positive solution h(z). For that it is necessary and sufficient that for each i both functions β_{i}(σ; z) and β_{i}(σ^{∗}; z) considered as functions of z are of one sign on Z. In its turn, for that it is necessary and sufficient that the interval (σ, σ^{∗}) on ℝ contains no points of 2ℤ. It gives is true if and only if σ belongs to the interval (−n, 2 − n) with n even. □

Now let us turn to the reducible case: σ ∈ 2ℤ. From (7.2) we see that if h(z) is defined outside some barrier, then h(z) = 0 inside of it. Therefore, we have to look for invariant positively definite Hermitian forms on irreducible subfactors. Omitting rather tiresome treatments, let us give the result.

Theorem 7.3. Here is the list of unitarizable irreducible subfactors of T_{σ}:

(a) the first discrete series: the representations on the subfactors V/V_{σ,3} for σ > 1 − n, σ ∈ 2ℤ; they are equivalent to the representations on the subspaces V_{σ,2} for σ < 1 − n, σ ∈ 2ℤ;

(b) the second discrete series: the representations on the subfactors V/V_{σ,2} for σ > 1 − n, σ ∈ 2ℤ; they are equivalent to the representations on the subspaces V_{σ,2} for σ < 1 − n, σ ∈ 2ℤ;

(c) the exceptional representation (lying on the boundary of the complementary series): it acts on the subspace V_{2} ∩ V_{3} for σ = 2 − n, n even, and, equvivalently, on the subfactor V/(V_{2} + V_{3}) for σ = −n, n even.

(d) the unit representation: it acts on the subspace V_{1} for σ = 0, and on the subfactor V/V_{4} for σ = 2 − 2n.

Notice that the representations on V_{2} and on V_{3} for σ = 1 − n, n odd, could be include in the first and the second discrete series respectively.

References

1. M. Berger. Les espaces symetriques non compacts. Ann. Sci. Éc. Norm. Supér., 1957, v. 74, 85–177.

2. A. Erdélyi et al. Higher Transcendental Functions. v. I, II. McGraw-Hill, New York, 1953, 1955.

3. J. Faraut. Distributions sphériques sur les espaces hyperboliques. J. Math. pures et appl., 1979, v. 58, No. 4, 369–444.

4. R. E. Howe, E.-Ch. Tan. Homogeneous functions on light cones: the infinitesimal structure of some degenerate principle series representations. Bull. Amer. Math. Soc., 1993, v. 28, No. 1, 1–74.

5. A. U. Klimyk. Matrix Elements and Clebsch-Gordan Coefficients of Group Representations. Naukova Dumka, Kiev, 1979.

6. A. U. Klimyk, B. Gruber. Structure and matrix elements of the degenerate series representations of U(p + q) and U(p, q) in a U(p) × U(q) basis. J. Math. Phys., 1982, v. 23, No. 8, 1399–1408.

7. V. F. Molchanov. Representations of a pseudo-orthogonal group associated with a cone. Mat. Sbornik, 1970, v. 81, No. 3, 358–375. Engl. transl.: Math. USSR Sbornik. 1970, v. 10, 333–347.

8. V. F. Molchanov. Representations of a pseudo-unitary group associated with a cone. In: Funkts. Anal., Ulyanovsk, 1984, issue 22, 55–66.

9. V. F. Molchanov. Harmonic analysis on homogeneous spaces. In: Itogi nauki i tekhn. Sovr. probl. matem. Fundam. napr. v. 59, VINITI, 1990, 5–144. Engl. transl.: Enc. Math. Sci. Springer, Berlin, 1995, v. 59, 3–136.

10. V. F. Molchanov. Maximal degenerate series representations of universial covering of the group SU(n, n). In: Lie Groups and Lie Algebras, Eds: B.P. Komrakov et al, Kluwer Acad. Publ., 1998, 313-336.

11. V. F. Molchanov , Yu. A. Sharshov. Representations of the universal covering group of the group SU(1, q) associated with a cone. Preprint Univ. Leiden, Report W97-09, 1997, 22p.

12. N. Ya. Vilenkin. Special Functions and the Theory of Group Representations. Nauka, Moscow, 1965. Engl. transl.: Transl. Math. Monographs 22, Amer. Math. Soc., Providence. 1968.

13. E. Wigner. Group Theory, Acad. Press, New York and London, 1959.

14. J. A. Wolf. Spaces of Constant Currature, Univ. Calif., Berkley, 1972.

15. D. P. Zhelobenko. Compact Lie Groups and Their Representations, Nauka, Moscow, 1970. Engl. transl.: Amer. Math. Soc., Providence, RI, 1973.

Received July 27, 1999