Integrate $x^2 e^{-x^2/2}$

By the Feynman trick we have:

$$I = \lim_{a\to 1}\int_0^{+\infty} -2\left(\frac{\text{d}}{\text{d}a} e^{-(a x^2)/2}\right)\ \text{d}x = \lim_{a\to 1}-2\frac{\text{d}}{\text{d}a}\int_0^{+\infty} e^{-(ax^2)/2}\ \text{d}x = \lim_{a\to 1} -2 \frac{\text{d}}{\text{d}a}\sqrt{\frac{\pi}{2a}}$$

Hence

$$I = \lim_{a\to 1}-2\left(-\frac{1}{2} \sqrt{\frac{\pi }{2}} \left(\frac{1}{a}\right)^{3/2}\right)$$

And our integral is simply

$$I = \sqrt{\frac{\pi }{2}}$$

Which is the result of your integral.

Hint: $u = x$, $dv = xe^{-x^2/2}dx$

No tricks, just the Gamma integral: Substituting $x = \sqrt{2t}$ gives $$\int_0^\infty x^2 e^{-x^2/2} \,dx = \sqrt{2\vphantom{X}} \int_0^\infty t^{1/2} e^{-t} \,dt = \sqrt{2\vphantom{X}}\,\Gamma\Bigl(\frac32\Bigr) = \sqrt{\frac\pi2}.$$