Non-orthogonal invariant subspaces

Let $T\in O(V)$ be an orthogonal matrix where $\dim V>0$. Write its characteristic polynomial $P_T$ as $$P_T=\det(XI-T)=\prod_{i=1}^k P_i^{m_i},$$ where the $P_i$ are distinct irreducible factors and the $m_i>0$ their multiplicities. Because $P_T$ is orthogonal it is diagonalizable, hence its minimal polynomial is $\prod_{i=1}^kP_i$.

For each $i$ define $U_i:=\ker P_i$ and let $T_i$ denote the restriction of $T$ to $U_i$. Then for each $i$ the minimal polynomial of $T_i$ is precisely $P_i$.

Proposition: The $U_i$ are pairwise orthogonal $T$-invariant subspaces and $V=\bigoplus_{i=1}^k U_i$.

Proof. See proposition 4.7 of the linked reader.

For every $u\in V$ let $U_u$ denote the subspace generated by the set $\{T^k(u):\ k\geq0\}$, where $T^0:=I$. Then $U_u$ is the smallest $T$-invariant subspace containing $u$. It is clear that

1. If $U$ is a $T$-invariant subspace and $u\in U$, then $U_u\subset U$.
2. If $U$ is an irreducible $T$-invariant subspace and $u\in U$ is non-zero, then $U_u=U$.

Because the $P_i$ are pairwise coprime, if a $T$-invariant subspace $U$ contains some element $$u=\sum_{i=1}^ku_i \qquad\text{ with }\ u_i\in U_i\ \text{ for each }1\leq i\leq k,$$ then it also contains $u_i$ for each $1\leq i\leq k$, and hence it contains the $T$-invariant subspace $U_{u_i}\subset U_i$. It follows that every irreducible $T$-invariant subspace is a subspace of some $U_i$. Because the $U_i$ are pairwise orthogonal it follows that non-orthogonal irreducible $T$-invariant subspaces are subspaces of the same $U_i$ for some $i$.

So let $U_1$ and $U_2$ be two non-orthogonal irreducible $T$-invariant subspaces of $U$ with $U_1\cap U_2=0$. Then without loss of generality the minimal polynomial of $T$ is an irreducible polynomial $P$.

For $u\in U$ let $T_u$ denote the restriction of $T$ to $U_u$. Then $P(T_u)=0$ so the minimal polynomial of $T_u$ divides $P$. But $P$ is irreducible, so the minimal polynomial of $T_u$ is also $P$ (unless $u=0$, then it is $1$). This implies that $\dim U_u=\deg P$, and hence every non-zero irreducible $T$-invariant subspace has dimension $\deg P$. In particular $\dim U_1=\dim U_2=\deg P$, proving the first statement.

The second statement holds if $d=1$ but fails if $d>1$:

If $d=1$ then for every $1$-dimensional subspace $U\subset U_1\oplus U_2$ we have $U=\langle u\rangle=U_u$ for every non-zero $u\in U$. This shows that every $1$-dimensional subspace of $U_1\oplus U_2$ is $T$-invariant, and of course it is irreducible.

For $d=2$ this fails; a counterexample for $n=4$ is given by the matrix $$T:=\begin{pmatrix} 0&-1&0&0\\ 1&\hphantom{-}0&0&0\\ 0&0&0&-1\\ 0&0&1&\hphantom{-}0 \end{pmatrix},$$ with the non-orthogonal irreducible $T$-invariant subspaces $$U_1:=\langle e_1,e_2\rangle \qquad\text{ and }\qquad U_2:=\langle e_1+e_3,e_2+e_4\rangle.$$ Here, the $2$-dimensional subspace $\langle e_3,e_4\rangle\subset U_1\oplus U_2$ is not $T$-invariant.

What is true, is that for every $u\in U_1\oplus U_2$ the subspace $U_u$ is irreducible and $T$-invariant, and moreover that every (non-zero) irreducible $T$-invariant subspace of $U_1\oplus U_2$ is of this form.

Note that $d>2$ does not occur over the real numbers, because there are no irreducible polynomials $P\in\Bbb{R}[X]$ with $\deg P>2$.

3. There are two orthogonal $d$-dimensional irreducible invariant subspaces $\bar U_1,\bar U_2\subset U_1\oplus U_2$.

This is true, and even stronger; there exists a $d$-dimensional $T$-invariant subspace $U_2'\subset U_1\oplus U_2$ that is orthogonal to $U_1$. For $d=1$ this is easier to see:

If $d=1$ then for $i\in\{1,2\}$ let $u_i\in U_i$ with $||u_i||=1$. Then $U_i=\langle u_i\rangle$, and setting $$u:=u_1-\frac{1}{\langle u_1,u_2\rangle}u_2 \qquad\text{ yields }\qquad \langle u_1,u\rangle=0,$$ so $U_u=\langle u\rangle$ is such a subspace.

For $d=2$ I think we can imitate the construction for $d=1$ with some adjustments, but I haven't got a proof (yet).

Allow me to translate this into more common representation-theoretic language. Saying that you have a finite subgroup $\Gamma \subseteq O(\mathbb{R}^n)$ is the same as the following data:

1. A finite group $G$,
2. A finite-dimensional real vector space $V$ equipped with a representation $\rho: G \to \operatorname{GL}(V)$, and
3. An inner product $\langle -, - \rangle: V \times V \to \mathbb{R}$ which is $G$-invariant, in the sense that $\langle \rho(g) v, \rho(g) u \rangle = \langle v, u \rangle$ for all $g \in G$ and $u, v \in V$.

Let $\{I_\lambda \mid \lambda \in \Lambda\}$ be a complete set of irreducible real representations of $G$. Simply knowing that $V$ is a real representation means that there is a canonical decomposition of $V$ into isotypic components, $V = \bigoplus_{\lambda} V_\lambda$, where $\lambda$ ranges over some indexing set for the isomorphism classes of irreducible representations of $G$. Here the subspace $V_\lambda$ is defined as the sum of all subrepresentations of $V$ isomorphic to $I_\lambda$. What is interesting is that these $V_\lambda$ must be orthogonal to each other.

Lemma: Suppose that $U, W \subseteq V$ are irreducible representations, and $\langle U, V \rangle \neq 0$. Then $U \cong V$ as real representations.

Proof: Since $\langle U, V \rangle \neq 0$, the map $\phi: U \to V^*, \phi(u)(v) = \langle u, v \rangle$ is nonzero. Furthermore, the $G$-invariancy of the inner product ensures that $\phi$ is a map of representations. Since $V \cong V^*$ as representations, we have found a nonzero $G$-equivariant map $U \to V$. By Schur's lemma, $U \cong V$.

This lemma shows that all the isotypic components $V_\lambda$ must be orthogonal under the $G$-invariant inner product. The answers to the rest of your questions basically follow from knowing that decomposition:

Answers to questions: let $U_1, U_2$ be irreducible subrepresentations of $V$, such that $\langle U_1, U_2 \rangle \neq 0$. Then:

1. $\dim U_1 = \dim U_2$, since by the above they must be isomorphic representations.
2. Every other nonzero $G$-invariant ($\dim U_1$)-dimensional subspace of $U_1 \oplus U_2$ must be isomorphic to $U_1$ as a representation, and hence irreducible. ($U_1 \oplus U_2$ is still inside the isotypic component).
3. The orthogonal complement of $U_1$ inside $U_1 \oplus U_2$ will be a subrepresentation which is both isomorpic and orthogonal to $U_1$.