Why is $\int x^2e^{x^2}\,d(x^2)$ not proper notation?

Students can feel more comfortable when they see what is substituted as a single entity. But of course

$$\int x^3e^{x^2}dx=\frac12\int x^2e^{x^2}d(x^2)=\frac12\int ue^{u}du$$

are equivalent and technically correct.

You can even do the integral mentally if you want (but make sure your instructor believes you).

---

Extra trick:

Write down $x^2$ on a label and read it loud "$u$".

The notation is proper. The Riemann-Sieltjes for a real-valued function $f(x)$ for $x\in [a,b]$ with respect to a real-valued function $g(x)$ is written

$$I=\int_a^b f(x) dg(x)$$

If $g$ is differentiable on $[a,b]$, then $I$ can be written as

$$I=\int_a^b f(x)\frac{dg(x)}{dx}\,dx$$

However, $g(x)$ need not be differentiable and, in fact, can even be discontinuous.

In the case of interest, $g(x)=x^2$ and $f(x)=x^2e^{x^2}$. We have, therefore

$$\int_a^b x^2e^{x^2}d(x^2)=2\int_a^b x^3 e^{x^2}\,dx$$