Solving a non-zero quadratic equation of two variables

Multiply the first equation by $x^2$ and substitute $xy$ from the second:

$$x^4-x^2y^2-3x^2=x^4-4-3x^2=0.$$

This is a biquadratic equation, which gives the roots $x^2=4$ and $x^2=-1$. If you are only interested in the real solutions,

$$x=\pm2,y=\frac2x.$$

$$\begin{cases} x^2 - y^2 = 3 \\ xy = 2, \end{cases}$$

Substitution is as follows, you substitute $y$ in first equation with it's value according to second equation so in this case. $$x^2-\frac4{x^2}=3$$ That is $y=2/x$ according to second equation. Substitution should always lead to reduction in number of variables.

Solving $x^2-\frac4{x^2}=3$ for $x$ yields $$x^4 -3x^2-4=0.$$ This equation has two real solutions $x=2$ and $x=-2$ (there are also two complex solutions at $x=i$ and $x=-i$). Plugging $x= \pm 2$ into the initial equations yields $y=\pm 1$. Likewise, you can obtain the complex solutions of $y$ by plugging $x=\pm i$ into the initial equations which yields $y=\pm 2i$.

An idea for you:

Multiply second equation by two: $\;2xy=4\;$ , and subtract it from first equation, getting:

$$x^2-2xy-(y^2-1)=0$$

The solution to this quadratic in $\;x\;$ is:

$$\Delta=4y^2+4y^2-4=4(2y^2-1)\implies x_{1,2}=y\pm\sqrt{2y^2-1}$$

and now substitute in first equation, say:

$$\overbrace{y^2\pm2y\sqrt{2y^2-1}+2y^2-1}^{=x^2}-y^2=3\implies y^2\pm y\sqrt{2y^2-1}=2\implies$$

$$y^4-4y^2+4=2y^4-y^2\implies y^4+3y^2-4=0\iff (y^2+4)(y^2-1)=0\implies$$

$$y=\pm1\implies x=\pm1\pm\sqrt{2-1}=\pm1\pm1$$

and since the solutions $\;(x,y)\;$ must have the same sign and both are clearly different from zero, we finally get the candidates:

$$(2,1)\,,\,\,(-2,-1)$$

Finally, check whether the above two really are solutions (since we squared stuff here...)