What is wrong with this proof that symmetric matrices commute?

You have proved that $v\mapsto v^TABv$ and $v\mapsto v^TBAv$ are the same quadratic form. However, since $AB$ and $BA$ are not necessarily symmetric, that doesn't mean they are the same matrix.

You can check this by plugging in some matrices where you know commutativity fails, for example $$ A =\begin{pmatrix} 1 & 0 \\ 0 & 2 \end{pmatrix} \quad B = \begin{pmatrix}0 & 1 \\ 1 & 0 \end{pmatrix} $$ We then get $$ AB = \begin{pmatrix}0 & 1 \\ 2 & 0 \end{pmatrix} \qquad BA = \begin{pmatrix}0 & 2 \\ 1 & 0 \end{pmatrix} $$ and indeed these define the same quadratic form: $$ (x\;\;y)\begin{pmatrix}0 & 1 \\ 2 & 0 \end{pmatrix}\begin{pmatrix}x\\ y\end{pmatrix} = 3xy = (x\;\;y)\begin{pmatrix}0 & 2 \\ 1 & 0 \end{pmatrix} \begin{pmatrix}x\\ y\end{pmatrix} $$ for all $x,y$, but the matrices are different.


With the same reasoning you can prove that for any square matrix $A$ it holds that $$\langle Ax,x\rangle =\langle A^T x,x\rangle.$$ You can't conclude that $A=A^T$ from here (you could if you would have $x$ and $y$ but that does not work).

So the thing is that the quadratic form $x\mapsto \langle Ax,x\rangle$ only determines the symmetric part of the matrix, but not the full matrix.


The fallacy in your proof is, as others have observed, in the last step, in which you go from the (correct) equation

$\langle \vec{v}, AB\vec{v} \rangle = \langle \vec{v}, BA\vec{v} \rangle$

to the incorrect conclusion

This holds true for any real vector $\vec{v}$ so therefore $AB = BA$

To see why this is wrong, let's rephrase the underlying assumption here as a question:

Suppose $M, N$ are two matrices such that for all vectors $\vec{v}$, $\langle \vec{v}, M\vec{v} \rangle = \langle \vec{v}, N\vec{v} \rangle$. What can you conclude?

You would like to conclude that $M = N$. Is that true?

Because the inner product is bilinear, the assumption that $\langle \vec{v}, M\vec{v} \rangle = \langle \vec{v}, N\vec{v} \rangle$ is equivalent to $\langle \vec{v}, (M-N)\vec{v} \rangle = 0$. If we introduce a new matrix $K = M-N$, then our question is equivalent to the following:

Suppose $K$ is a matrix such that for all vectors $\vec{v}$, $\langle \vec{v}, K\vec{v} \rangle = 0$. What can you conclude about $K$?

Again, you would like to conclude that $K = 0$. Is that necessarily true?

Stated in this form it is perhaps easier to see where you have made your mistake. It is true that $\langle \vec{v}, \vec{v} \rangle = 0 \Longrightarrow \vec{v} = 0$ (this is one of the properties of any inner product), and the property you wish to assert looks sort of like this, if you don't look too closely. But it is not true in general that if $\langle \vec{v}, K\vec{v} \rangle = 0$ for all $\vec{v}$, then $K = 0$. Indeed, $\langle \vec{v}, K\vec{v} \rangle = 0$ holds for any antisymmetric matrix $K$. In the example of the original question $K = AB - BA$, the commutator of the two given symmetric matrices, which is always antisymmetric.