Prove that $\det(AB - BA) = \frac{1}{3}\left(\mathrm{Trace}(AB - BA)^3\right)$

This follows easily from Cayley-Hamilton theorem. Since $M=AB-BA$ has zero trace, by Cayley-Hamilton theorem, $M^3=cM+dI_3$ where $c$ is some scalar and $d=\det(M)$. Therefore $\operatorname{tr}(M^3)=3d$ and the result follows.


Choose a basis which puts $AB-BA$ in Jordan normal form (both $\det$ and $\text{Tr}$ are invariant under basis change, so this is allowed). Then since a commutator is traceless, the diagonal must have the form $(a,b,-a-b)$. Then $\det(AB-BA)=-ab(a+b)$, and $$\frac{1}{3}\text{Tr}(AB-BA)^3=\frac{1}{3}(a^3+b^3-(a+b)^3) = -ab(a+b)$$ as desired.


Wikipedia gives the following identity for the determinant of a $k\times k$ matrix: $$\det A= \frac{1}{k!} \det\begin{pmatrix} \operatorname{tr}A & k-1 &0&\cdots & \\ \operatorname{tr}A^2 &\operatorname{tr}A& k-2 &\cdots & \\ \vdots & \vdots & & \ddots & \vdots \\ \operatorname{tr}A^{k-1} &\operatorname{tr}A^{k-2}& & \cdots & 1 \\ \operatorname{tr}A^k &\operatorname{tr}A^{k-1}& & \cdots & \operatorname{tr}A \end{pmatrix}. $$ In particular when $k=3$, and using $AB-BA$ in place of $A$, we have that $$ \det (AB-BA)=\frac{1}{3!}\det\begin{pmatrix} 0 & 2 &0 \\ \operatorname{tr}(AB-BA)^2 &0& 1 \\ \operatorname{tr}(AB-BA)^{3} &\operatorname{tr}(AB-BA)^{2}& 0 \\ \end{pmatrix}. $$ Expanding the determinant along the top row yields your formula.