Find all solutions to the equation $x^2 + y^2 + xy = (xy)^2$

You can write your equation (leaving the trivial $x=0,y=0$) as $$\frac xy+ \frac yx +1 =xy$$ now let $x/y=k$ and rearrange: $$k^2(1-y^2)+k+1=0$$ which has rational solutions only if $4y^2-3=p^2$. This implies that $y=\pm 1$. Hence the only integer solutions are $(-1,1)$ and $(1,-1)$.

I'm assuming you're looking for integer solutions, even though you didn't say os.

If you rewrite the LHS as $(x+y)^2 - xy$, your equations becomes $$(x+y)^2 = xy(xy + 1)$$

For positive $x$ and $y$:

Suppose (by swapping names if necessary) that $x \ge y$. Then the left hand side is no more than $4x^2$, while the right hand side is at least $y^2 x^2$.
We have $$ LHS = (x+y)^2 \le (x+x)^2 = 4x^2 $$ and $$ RHS = xy(xy + 1) > (xy)^2 = y^2 x^2. $$ If $y \ge 2$, then $$ RHS > y^2 x^2 \ge 2^2 x^2 = 4 x^2 $$ but RHS and LHS are equal, so we have a number that's both $ \le 4x^2$ and $> 4x^2$, and that's a contraduction.

That means that $y$ is at most one. That should get you on your way.

For the case where one of $x$ and $y$ is negative, you still have some work to do.

First, write it as $(x^2-1)(y^2-1)=xy+1$.

In the narrow ranges where $|x|\leq 1$ or $|y|\leq 1$, you can solve yourself.

Then, when $x>1$ and $x>1$ you have $(x^2-1)(y^2-1)>=2(x^2+y^2-2)=2(x-y)^2+4xy-4\geq 4xy-4$.

But $xy\geq 4$, so $4xy -4> xy +3\cdot 4 -4 \geq xy-1$.

The case when $x<-1$ and $y<-1$ is the same.

There are no cases when $x<-1$ and $y>1$ because then $xy+1<0$.

Finally, if $x=0$, $y=0$. If $x=-1$ then $y=1$, and if $x=1,$ $y=-1$.

The key is that in most cases, it is "obvious" that $(x^2-1)(y^2-1)$ is a lot bigger than $xy$.

---

Another approach. Let $d=xy$. Then assume $d\neq 0$ and you have:

$$x^2+\frac{d^2}{x^2}=d^2-d$$

or

$$x^4-(d^2-d)x^2 + d^2=0$$

So by the quadratic formula:

$$x^2=\frac{d^2-d \pm \sqrt{d^2(d-3)(d+1)}}{2}$$

When is $(d-3)(d+1)=(d-1)^2-4$ a perfect square? The only case of perfect squares that differ by four is $0$ and $4$. That means $d=3$ or $d=-1$.

The only cases left to handle then are $d=0$ and $d=3$.