Is it true that ${\rm Tr}((A^{1/2}BA^{1/2})^{1/2}) = {\rm Tr}((BA)^{1/2})$ for positive semidefinite matrices $A,B$?

The short answer is no. A matrix $M$ in general possesses more than one (or even infinitely many) square roots. When $M$ is positive semidefinite, $M^{1/2}$ is conventionally defined as the unique positive semidefinite square root of $M$. However, when $M$ isn't positive semidefinite, you need to be careful about what $M^{1/2}$ refers to.

E.g. when $A=\pmatrix{1&0\\ 0&4}$ and $B=\pmatrix{8&6\\ 6&5}$, their product $BA=\pmatrix{8&24\\ 6&20}$ is not positive semidefinite (it isn't symmetric in the first place). Note that both $X=\pmatrix{2&4\\ 1&4}$ and $Y=\frac{1}{\sqrt{5}}\pmatrix{2&12\\ 3&8}$ are square roots of $BA$ (i.e. $X^2=Y^2=BA$), but $\operatorname{tr}(X)=6\ne\sqrt{20}=\operatorname{tr}(Y)$.

You can make the answer positive, however, if we take $(BA)^{1/2}$ as any square root that has nonnegative eigenvalues. In this case, since $BA$ is similar to $A^{1/2}(BA)A^{-1/2}=A^{1/2}BA^{1/2}$, we have $$ \lambda_i\left((BA)^{1/2}\right)=\sqrt{\lambda_i(BA)}=\sqrt{\lambda_i(A^{1/2}BA^{1/2})}=\lambda_i\left((A^{1/2}BA^{1/2})^{1/2}\right) $$ and hence $(BA)^{1/2}$ and $(A^{1/2}BA^{1/2})^{1/2}$ have identical eigenvalues and identical traces.


If $U$ and $V$ are invertible, then $$\det(UV-\lambda I) = \det(U) \det(V - \lambda U^{-1}) = \det(V - \lambda U^{-1}) \det(U) = \det(VU-\lambda I) .$$ So $UV$ and $VU$ have the same eigenvalues by multiplicity. By taking limits (that is, replace $U$ and $V$ by $U + \mu I$ and $V+\mu I$ and let $\mu \to 0$), we can see that this is true even if $U$ and $V$ are not invertible.

So if $U = A^{1/2}$, and $V = B A^{1/2}$, we see that the sum of the square roots of the eigenvalues of $UV$ are the same as that for $VU$.