Does $AB+BA=0$ imply $AB=BA=0$ when $A$ is real semidefinite matrix?
Let $v$ be an eigenvector of $A$ with eigenvalue $\lambda$. Then $$ 0 = A(Bv) + B(Av) = A(Bv)+\lambda Bv\Longrightarrow A(Bv) = -\lambda Bv $$ what this means is that $Bv$ is also an eigenvector of $A$ with eigenvalue $-\lambda$. But since $A$ is positive semi-definite all eigenvalues of $A$ are $\geq 0$. If $\lambda=0$, so we have two possiblities:
- Either $v$ is an eigenvector with eigenvalue zero, in which case $A(Bv) = 0$, This means $ABv = BAv = 0$ for all eigenvectors corresponding to eigenvalue zero.
- Or $v$ is an eigenvector with eigenvalue $\lambda>0$. Suppose $Bv\neq 0$, then $Bv$, by above observation is an eigenvector of $A$ with negative eigenvalue! This is impossible, so $Bv=0$. Therefore no only $ABv = 0$, we also have $BAv = \lambda Bv = 0$.
Since all vectors can be written as a linear composition of these eigenvectors, we have $AB=BA = 0$ in these circumstances.
Extra: If you are also interested in what this $B$ can actually be if $AB+BA = 0$, note that if $w$ is any vector, and $P_0$ is the porjection linear transformation which sends $w$ to the subspace of vectors with eigenvalue zero. Then we know the following:
Write $w = P_0w + (1-P_0)w$, then by second observation above $B(1-P_0)w = 0$. So in the basis in which $A$ is diagonal (since $A$ is symmetric it is diagonalizable), $B$ is necessarily of the form $$ B=\begin{pmatrix} B_0 & 0\\ 0 & 0 \end{pmatrix} $$ In other words $B$ satisfies $AB+BA=0$ if and only if $P_0BP_0=B$.
Proposition. Let $A,B\in M_n(\mathbb{C})$; if $A$ is diagonalizable with non-negative eigenvalues and $AB+BA=0$, then
$AB=BA=0$.
If moreover $A$ is hermitian, then $A,B$ are unitarily simultaneously similar to $A_1=diag(D_p,0_{n-p}),B_1=\begin{pmatrix}0_p&0\\0&T_{n-p}\end{pmatrix}$, where $D$ is diagonal $> 0$ and $T$ is upper triangular.
Proof. For 1. cf. the Hamed's answer.
For 2. Using the syeh_106's proof, $A,B$ are unitarily simultaneously similar to $A_1=diag(D_p,0_{n-p})),B'=\begin{pmatrix}0_p&0\\0&U_{n-p})\end{pmatrix}$, where $D$ is diagonal $> 0$. It remains to triangularize $U$, using Schur's method.