Why does the limit definition of e fail?
Your first approach uses the reasoning
$(1+\frac1x)^x$ is approximately $e$, so $(1+\frac1x)^{x^2}=[(1+\frac1x)^x]^x$ behaves like $e^x$.
This is illegal because you're letting $x\to\infty$ in your inner expression while preventing $x\to\infty$ in the outer expression. There's no rule that allows you to do that -- you can't hold $x$ fixed in one part of your expression while letting it tend to infinity in another part.
As a simpler example, it is incorrect to compute $\lim_{n\to\infty}1$ by writing $1=\frac1n\cdot n$, then observing that $\frac1n\to0$ as $n\to\infty$, and deducing that $$ \lim_{n\to\infty}1 = \lim_{n\to\infty}\left[ \left(\lim_{n\to\infty}\frac 1n\right)\cdot n \right]= \lim_{n\to\infty}(0\cdot n) = 0. $$
Another simpler, but relevant example: $(1+\frac1x)$ tends to $1$ as $x\to\infty$, but it doesn't follow that $$\lim_{x\to\infty}\left(1+\frac1x\right)^x = \lim_{x\to\infty}\left[\lim_{x\to\infty}\left(1+\frac1x\right)\right]^x=\lim_{x\to\infty} 1^x = 1. $$
The short answer is that $\lim_{x \to \infty} g(x, f(x)) \ne \lim_{x \to \infty}g (x, \lim_{x \to \infty} f(x))$. (Here, $g(x,y) = \frac{y^x}{e^x}$ and $f(x)=(1+1/x)^x$.) See grand_chat's answer for a very stark example of why taking limits of parts of the function before taking the limit as a whole is "illegal."
The long answer is that although $(1+1/x)^x$ does get close to $e$ as $x \to \infty$, you cannot simply replace it with $e$ (before taking the limit again) because you don't know how the discrepancy between $e$ and $(1+1/x)^x$ will interact with the power of $x$ on the outside and the rest of the expression.
$$((1+1/x)^x)^x = [e + \underbrace{((1+1/x)^x - e)}_{\text{error}}]^x$$
In principle, you have to understand the asymptotics of the error term. If the error vanishes extremely quickly, it might be ok to ignore it. However, as you see in this example, this [negative] error term is significant enough that it "slows down" the numerator to $e^{x-1/2}$ in some sense.
Long story short, make sure each step you take is a valid application of a limit property.