In quantum mechanics, why is $\langle T\rangle=\frac{\langle p^2 \rangle}{2m}$ rather than $\langle T\rangle=\frac{\langle p \rangle^2}{2m}$?

If you think about it, this really doesn't come down to QM and just depends on how you take averages. QM only comes into play if you actually want to calculate those averages given the state vector of the system.

We know that $T=\frac{p^2}{2m}$, so the average of this is then $$\langle T\rangle=\left\langle\frac{p^2}{2m}\right\rangle=\frac{\langle p^2\rangle}{2m}$$

Since, in general, $\langle p^2\rangle\neq\langle p \rangle^2$, this is where we end up.

If you want to find this value using the position basis, then we invoke QM: $$\langle T\rangle=-\frac{\hbar^2}{2m}\int\Psi^*\frac{\partial^2}{\partial x^2}\Psi\ dx$$

This is because in the position basis, the $P^2$ operator is $-\hbar^2\frac{\partial^2}{\partial x^2}$.


Why is this?

For concreteness, let's look at a specific example for which $\langle T \rangle \ne \frac{\langle P \rangle^2}{2m}$

Consider the case that we have a particle with state vector (working in 1D for simplicity)

$$|\psi\rangle = \frac{1}{\sqrt{2}}\left(|+p\rangle + |-p\rangle\right)$$

where $p \ne 0$ and $P\,|\pm p\rangle = \pm p\,|\pm p\rangle$ (these are eigenkets of the momentum operator).

Clearly, the expectation value of momentum is

$$\langle P\rangle = \langle\psi|P|\psi\rangle = \frac{1}{2}\left(+p -p\right) = 0$$

This is because the momentum measurement has equal chance of yielding $+p$ and $-p$.

However, a kinetic energy measurement measurement can only yield

$$T = \frac{(\pm p)^2}{2m} = \frac{p^2}{2m}$$

and so

$$\langle T \rangle = \frac{p^2}{2m} \ne \frac{\langle P \rangle^2}{2m} = 0$$