A Diophantine equation solved when N is not a square?
Here's one approach, though possibly not the simplest.
A number $N$ is not a perfect square if $A^2+1 \le N \le (A+1)^2-1$ for some $A$. But how can we encode $X \le Y$? Over the real numbers, the standard trick would be to write $Y = X + Z^2$, because $Z^2$ is always nonnegative. Over the integers, that doesn't quite work, but we know that $X \le Y$ if and only if there are four integers $P,Q,R,S$ such that $Y = X + P^2 + Q^2 + R^2 + S^2$, by using Lagrange's four-square theorem.
So we get $$ N = A^2+1 + B^2 + C^2 + D^2 + E^2 \text{ and } A^2+2A = N + F^2 + G^2 + H^2 + I^2 $$ but we probably want to write this as a single equation. To do this, just take $$ (N - A^2-1-B^2-C^2-D^2-E^2)^2 + (A^2+2A-N-F^2-G^2-H^2-I^2)^2 = 0. $$ Over the integers (or even the reals), $X^2+Y^2=0$ only if $X=Y=0$, giving us what we wanted.
Actually, the above only characterizes positive nonsquares: if $N$ is negative, there's no value of $A$ we can choose. But we can multiply the equation above by an equation representing $N\le-1$ easily, fixing this issue.
Pell's equation $$X^2-NY^2=1$$ has a solution in integers other than the trivial $(\pm1, 0)$ if and only if $N$ is not a square.
To express that $X^2 \neq 1$, one can add one of the equations from What is the simplest Diophantine equation equivalent to N is not zero?
that is, one of:
- $Z=1+A^2+B^2+C^2+D^2$
- $AZ=(2B+1)(3B+1)$
- $AZ=7+B^2+BC+C^2$
where $Z=X^2-1$
Edit: as in Misha Lavrov's answer, this only characterizes positive non-squares.