What are the possible eigenvalues of these matrices?

There is a general framework for answering questions like this although I don't know the answer in this case. You are asking for which coadjoint orbits of $U(8)$ the moment polytope for the action of the subgroup $SU(4) \times SU(2) \times SU(2)$ contains the origin. There is a method for computing these polytopes in Berenstein, Arkady; Sjamaar, Reyer, Coadjoint orbits, moment polytopes, and the Hilbert-Mumford criterion, J. Am. Math. Soc. 13, No.2, 433-466 (2000). ZBL0979.53092..

There is a way of packaging all the coadjoint orbits together, by saying that each coadjoint orbit is a symplectic quotient of the cotangent bundle of the group. This rephrases the problem as that of computing the moment polytope for the $U(8)$ action on $T^*(U(8)/SU(4) \times SU(2) \times SU(2))$. This is going to be a convex polyhedral cone.

General theory says that you can compute the affine space spanned by the polytope from the generic stabilizer (the subgroup of elements fixing a generic point). In your case, the generic stabilizer is the same as the generic stabilizer of $SU(4) \times SU(2) \times SU(2)$ on the quotient of Lie algebras $\mathfrak{u}(8)/{\mathfrak{su}}(4) \oplus \mathfrak{su}(2) \oplus \mathfrak{su}(2)$.

In the 4 by 4 case you mentioned, to find the space perpendicular to the moment polytope you want to compute the generic stabilizer of $G = SU(2) \times SU(2)$ on $V = \mathfrak{u}(4)/\mathfrak{su}(2) \oplus \mathfrak{su}(2)$. Except for the multiples of the 2 by 2 identity this quotient can be identified with $2 \times 2$ matrices. The generic stabilizer is only defined up to conjugacy, and it's a bit tricky to find the equation for a moment polytope as opposed to one of its Weyl conjugates. The stabilizer $G_B$ at a matrix $B$ is the subgroup of $(A_1,A_2)$ so that $A_1 B A_2^{-1} = B$. Any $B$ is diagonal up to left and right multiplication, and so any diagonal $B$ with generic eigenvalues gives the generic stabilizer. But taking $B$ diagonal the resulting hyperplane doesn't meet the positive Weyl chamber; this is related to the fact that the answer that you want is going to depend on how the eigenvalues are ordered, so its better to take $B$ antidiagonal. (Once one accepts that the generic stabilizer is abelian, any full rank $B$ is ok.) Take $$B = \left[ \begin{array}{cc} 0 & b_{12} \\ b_{21} & 0 \end{array} \right] , \ |b_{12}| \neq |b_{21}|, \ \ A_2 = \left[ \begin{array}{ll} a_{11} & a_{12} \\ a_{21} & a_{22} \end{array} \right] .$$ Then we want $$A_1 = B A_2 B^{-1}$$ to be special unitary. Since $$ B A_2 B^{-1} = \left[ \begin{array}{ll} a_{22} & (b_{12}/b_{21}) a_{21} \\ (b_{21}/ b_{12}) a_{12} & a_{11} \end{array} \right] $$ is unitary only if $a_{12} = a_{21} =0$, we have $$A_2 = \operatorname{diag}(t^{-1},t), \ \ t= a_{22}, \ A_1 = \operatorname{diag}(t,t^{-1}) .$$ Hence the generic stabilizer is the set of matrices $$diag(t,t^{-1},t^{-1},t)$$ for some complex $t$ with norm one. The Lie algeba of the stabilizer is the span of $(1,-1,-1,1)$, and the perpendicular is the space of $(\lambda_1,\lambda_2,\lambda_3,\lambda_4)$ satisfying

$$\lambda_1 - \lambda_2 - \lambda_3 + \lambda_4 = 0 .$$

In the 8 by 8 case you want to understand, the quotient $\mathfrak{u}(8)/\mathfrak{su}(4) \oplus \mathfrak{su}(2) \oplus \mathfrak{su}(2)$ is (after forgetting about diagonal matrices) identified with the space of pairs $(A,B)$ where $A$ is 4 by 4 and $B$ is 2 by 2. Generically such matrices are full rank, and take $B$ to be a generic antidiagonal matrix implies that the the matrices in $SU(2) \times SU(2)$ must be of the form diag$(t,t^{-1},t^{-1},t)$. But then we want $$ A\,\text{diag}(t,t^{-1},t^{-1},t) A^{-1} $$
to be special unitary which is not the case for generic $A$. So the generic stabilizer has trivial Lie algebra, which means that the moment polytope is full rank (that is, no linear equations are satisfied).

In a previous version of this answer, I accidentally took $A$ to be generic "unitary" and got the wrong answer that the cone has codimension one. However, my previous wrong answer does suggest something about the facets of the cone. General theory says that the hyperplanes at the boundary of the cone are perpendicular to one-dimensional stabilizers. If one takes $A$ to be a permutation matrix then one gets an element with one-dimensional stabilizer. So I wonder whether the cone you are looking for is the cone whose facets are among those defined by equalities $$ \lambda_{\sigma(1)} - \lambda_{\sigma(2)} - \lambda_{\sigma(3)} + \lambda_{\sigma(4)} + \lambda_{\sigma(5)} - \lambda_{\sigma(6)} -\lambda_{\sigma(7)} + \lambda_{\sigma(8)} = 0 $$
where $\sigma$ ranges over elements of the eighth symmetric group. The Berenstein-Sjamaar paper would answer this with enough work. (It is a Schubert calculus computation.)


This is a partial answer that shows an obstruction to certain eigenvalue sequences. First, I claim that if $M$ is rank one then it isn't similar to something of the stated form.

Take a matrix $M$ of the given form and suppose the rank is $0$ or $1.$ If $a\neq0$, then the rank of $M$ is at least 4. If $b\neq0$ or $c\neq0$ then the rank of $M$ is at least 2. Since $M$ is rank 0 or 1 we must have $a=b=c=0.$ Since we have a 0 diagonal, if $A\neq0$, then the rank of $M$ would have to be at least 2 so we must have $A=0$. Similarly, the 0 diagonal and $B\neq0$ forces the rank to be at least 2 so we must have $B=0$, so $M$ is rank $0.$

That eliminates some eigenvalue sequences. Now notice that if the rank of $M$ is $\leq 3$ and $M$ is positive semidefinite this forces $a=0$ and $A=0$ so you are back in the case that you know how to deal with. This will eliminate some other eigenvalue sequences.


Consider the case where the $8\times 8$ matrix is positive semidefinite and assume that the 5 largest eigenvalues are all equal. Then by the argument of Federico Poloni they equal a. Then it follows $A = 0$ and therefore the four smallest eigenvalues are restricted like in the $4\times 4$ case .